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1.  Introduction 

This  report  describes  the  work  done  for  the  Naval  Post  Graduate  School  and  in 
association  with  Starfire  Optical  Range,  Kirtland  Air  Force  Base,  under  NPS  Grant 
N00244-07-1-009,  between  28  June  2007  and  31  October  2008. 

This  work  relates  to  the  Air  Force’s  need  to  characterize  the  cloud  distribution  during  day 
and  night,  for  a  variety  of  applications.  These  applications  include  support  of  research 
into  the  impact  of  clouds  on  laser  communication  and  support  of  satellite  tracking.  This 
contract  followed  Contract  N00014-01-D-0043  DO  #4,  which  is  documented  in  Shields 
et  al.  2007a,  Technical  Note  271,  Contract  N00014-01-D-0043  DO  #11,  which  is 
documented  in  Shields  et  al.  2007b,  Technical  Note  272,  and  Contract  N00014-01-D- 
0043  DO  #13,  which  is  documented  in  Shields  et  al.  2007c,  Technical  Note  273. 

Under  the  NPS  Grant,  we  finished  preparing  a  refurbished  Whole  Sky  Imager  (WSI), 
Unit  8,  and  fielded  it  at  Air  Force  Site  3.  This  completed  deployment  to  the  three  sites 
we  were  responsible  for,  Sites  2,  3,  and  5.  In  addition,  we  repaired  the  WSI  at  Site  5  after 
it  was  struck  by  lightning,  and  got  the  WSI  at  the  SOR  site  operational.  We  upgraded  the 
daytime  cloud  algorithm  to  include  an  adaptive  feature  to  adjust  for  haze  changes,  and 
upgraded  the  nighttime  cloud  algorithm  to  include  moonlight  conditions  in  the  new  high 
resolution  algorithm.  We  delivered  these  algorithms  so  they  were  running  in  real  time  at 
Sites  3  and  5,  and  also  processed  and  delivered  significant  amounts  of  archival  data  from 
these  two  sites. 

A  follow-on  Contract,  FA9451-008-C-0226,  was  funded  through  the  Air  Force  Research 
Laboratory  on  1  February  2008.  This  work  helped  fund  some  of  the  later  work 
mentioned  in  the  paragraph  above.  The  other  work  under  this  contract  will  be  reported 
under  a  separate  report. 

2.  Background 

A  series  of  digital  automated  Whole  Sky  Imagers  (WSI)  have  been  developed  by  MPL 
over  many  years,  beginning  in  the  early  1980’s  (Johnson  et  al.  1989  and  1991,  and 
Shields  et  al.  1993,  1994,  1997a  and  b).  (Published  references  are  listed  in  Section  12.3.) 
These  systems  are  designed  to  acquire  accurate  imagery  of  the  full  upper  hemisphere  in 
several  spectral  filters,  in  order  to  assess  the  presence  of  clouds  at  each  pixel  in  the 
image.  A  system  capable  of  24-hour  operation,  the  Day/Night  WSI,  was  developed  by 
MPL  under  funding  from  the  Air  Force,  Navy,  and  Army  in  the  early  1990’s  (Shields  et 
al.  1998,  2003b  and  e,  2004  a  and  b,  and  2005b  and  c).  One  of  the  first  two  units  was 
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fielded  at  the  Air  Force’s  Starfire  Optical  Range  in  October  1992.  Technical  Memo 
AV06-034t  documents  the  estimated  run-time  experience  with  the  different  systems. 
(Technical  Memos  are  listed  in  Section  12.1.)  We  have  approximately  75  system-years, 
or  650,000  hours  of  experience  with  the  Day/Night  WSI  systems  at  this  time. 

Related  systems  have  been  designed  and  fielded  over  the  years.  These  include  a  Daytime 
Visible/NIR  WSI  (Feister  et  al.  2000,  and  Shields  et  al.  2003d),  and  visible  and  Short¬ 
wave  IR  systems  for  airborne  use  (Shields  et  al.  2003c).  An  imaging  system  for 
measuring  visibility  was  developed  and  successfully  tested  (Shields  et  al.  2005a  and 
2006).  A  field  calibration  system  was  developed  to  enable  calibration  of  the  D/N  WSI  in 
the  field  (Shields  et  al.  2003a). 

SOR  has  funded  development  of  additional  instruments  and  more  advanced  algorithms, 
as  well  as  support  of  instruments  in  the  field,  for  several  years.  Under  Contract  N00014- 
97-D-0350  DO  #2  (September  1997  -  June  2001),  MPL  was  funded  to  develop  and 
provide  a  new  Day/Night  WSI,  which  was  designated  Unit  12  (Shields  et  al.  2003b). 
This  was  the  first  system  using  a  Windows  operating  system  and  the  Series  300 
Photometries  cameras.  We  also  analyzed  and  provided  statistical  estimates  of  Cloud  Free 
Line  of  Sight  (CFLOS)  and  related  properties. 

Under  Contract  N00014-97-D-0350  DO  #6  (May  1999  -  May  2003),  we  were  funded  to 
provide  two  upgraded  instruments,  Units  13  and  14  (Shields  et  al.  2004b).  These 
instruments  included  integration  of  the  control  computer  into  the  outdoor  enviromnental 
housing,  and  new  software  to  provide  near-real-time  cloud  processing  on  an  additional 
processing  computer.  In  addition,  we  developed  a  night  cloud  algorithm  based  on 
detection  of  the  contrast  between  the  signal  from  stars  and  their  background.  The 
concepts  had  been  developed  under  funding  from  another  sponsor  (Shields  et  al.  2002). 
Under  Delivery  Order  #6,  this  concept  was  expanded  to  handle  moonlight,  and  converted 
to  a  fieldable  C-code  and  installed  on  Unit  12  running  at  the  SOR  site. 

Under  Contract  N00014-01-D-0043  DO  #4  (May  2001  -  September  2006),  documented 
in  Shields  et  al.  2007a,  we  fielded  WSI  Unit  14  at  a  site  in  Virginia,  and  supported  this 
deployment  as  well  as  the  continued  operation  of  Unit  12  at  the  SOR  site.  Also  under 
DO  #4  the  day  algorithm  at  the  SOR  site  was  updated  to  run  in  real  time  (like  Units  13 
and  14)  and  include  a  horizon  and  occultor  mask,  and  the  clear  sky  background  was 
updated.  A  stand-alone  version  of  the  algorithm  was  written  for  in-house  data  analysis. 
Under  funding  from  another  sponsor,  an  extensive  data  base  was  processed  and  analyzed 
to  extract  CFLOS  statistics.  This  work  allowed  us  to  determine  the  strengths  and 
weaknesses  of  the  daytime  cloud  algorithm.  The  night  algorithm  was  further  developed 
to  handle  the  anthropogenic  light  sources  near  the  SOR  site.  The  new  night  algorithm 
was  installed  to  run  in  real  time  at  the  SOR  site. 

Under  DO  #4  we  also  provided  an  extensive  analysis  of  IR  systems  and  their  pros  and 
cons  for  this  program.  Most  of  this  analysis  concentrated  on  Long  Wave  IR  (LWIR) 
systems  in  the  8-12  pm  wavelength  region,  because  our  analysis  showed  that  Short 
Wave  IR  (SWIR)  systems  near  the  1.6  pm  wavelength  region  would  not  be  adequate  for 
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our  purposes,  and  Mid  Wave  IR  (MWIR)  systems  near  the  3  -  5  pm  wavelength  region 
had  disadvantages  with  respect  to  the  LWIR  systems.  The  LWIR  analysis  of  theoretical 
performance  and  images  we  had  access  to  show  that  the  LWIR  system  can  be  expected  to 
have  difficulty  in  environments  other  than  those  that  are  high  and  dry,  particularly  with 
thin  high  clouds  as  well  as  all  clouds  near  the  horizon.  This  analysis,  as  well  as  the  other 
work  done  under  DO  #4,  is  presented  in  Shields  et  al.  2007a. 

Contract  N00014-01-D-0043  DO  #11  (September  2004  -  April  2006),  is  documented  in 
Shields  et  al.,  2007b.  During  this  time,  we  received  some  of  the  older  DOS-based  WSI 
systems  retired  from  the  program  we  had  built  them  for,  and  began  refurbishing  them  for 
use  with  this  project.  The  refurbishment  also  included  creating  the  control  software  and 
the  processing  software  to  enable  these  older  hardware  systems  to  meet  the  needs  of  the 
SOR  program.  Many  new  features  were  added  to  the  software  during  this  time.  The  first 
of  the  refurbished  WSI’s  was  operational  in  June  05,  and  ran  well,  but  was  not  fielded 
because  the  site  was  not  ready. 

We  were  asked  to  direct  much  of  the  effort  under  DO  #11  toward  processing  and  testing 
data,  and  developing  methods  to  evaluate  the  results.  Program  SORCloud Assess  was 
developed  to  enable  testing,  and  day  data  from  the  SOR  site  were  processed  and  found  to 
be  very  good,  with  an  estimated  accuracy  of  about  98%.  This  program  does  not  use  a 
blind  test,  so  the  numbers  could  be  optimistic,  however  the  program  does  enable  us  to 
quantitatively  compare  different  sites  and  algorithm  versions. 

We  were  challenged  to  process  data  for  a  hazy  site,  so  data  from  the  Virginia  site  were 
processed,  and  the  day  cloud  algorithm  also  worked  well  for  this  site,  with  an  estimated 
accuracy  of  about  98%.  For  the  VA  data,  we  updated  the  day  algorithm  to  use  Near 
Infrared  (NIR)  data,  and  also  tested  an  adaptive  algorithm  concept. 

Meanwhile,  the  night  algorithm  data  were  tested  using  Program  SORCloudAssess.  The 
accuracy  is  more  difficult  to  assess  due  to  the  moderate  resolution  of  the  night  algorithm, 
however  we  still  got  results  of  about  95%  using  the  “region  of  sight”,  as  discussed  in 
Technical  Note  272.  We  also  developed  a  method  to  ground-truth  the  results  using  the 
brightest  stars,  and  developed  and  tested  beginning  concepts  for  a  high  resolution  night 
algorithm.  Much  of  the  work  to  convert  the  night  algorithm  to  a  transmittance-based 
algorithm  (rather  than  contrast-based)  was  done  under  this  contract.  Work  in  several 
other  areas,  including  evaluating  night  data  in  cities,  and  developing  forecasting 
techniques,  was  also  begun  under  this  contract. 

Contract  N00014-01-D-0043  DO  #13  (April  2006  -  July  2007),  is  documented  in  Shields 
et  al.,  2007c.  Under  this  contract,  we  fielded  the  WSI  that  was  refurbished  under  DO 
#11,  fielding  it  at  Site  2  in  May  2006.  A  second  WSI  was  refurbished  and  deployed  to 
Site  5  in  January  2007,  and  a  third  WSI  was  partially  refurbished.  The  software  was 
upgraded  in  several  ways,  and  also  the  QC  capabilities  and  software  were  more  fully 
developed. 
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Under  DO  #13,  the  field  version  of  the  day  algorithm  was  upgraded  to  allow  using  the 
Near  Infrared  (NIR)  data,  which  often  does  a  better  job  in  heavy  haze.  This  was  fielded 
at  Site  2.  The  algorithm  did  not  yet  include  a  feature  for  adjusting  for  the  variations  in 
haze  amount,  but  even  just  the  conversion  to  the  NIR  was  a  significant  improvement. 

More  importantly,  we  developed  a  new  night  algorithm  based  on  the  earth-to-space 
transmittance  measured  at  star  locations,  which  provided  high  resolution  cloud  results  at 
night.  Although  we  did  not  have  time  under  this  contract  to  optimize  the  algorithm  for 
moonlight  conditions,  it  worked  very  well  under  starlight  conditions.  Details  of  these 
accomplishments,  as  well  as  others,  are  documented  in  Technical  Note  273.  DO  #13  was 
the  contract  that  immediately  preceded  the  grant  under  discussion  in  the  current  report. 

The  primary  goals  of  the  NPS  Grant  were  to  complete  the  refurbishment  and  deployment 
of  the  third  WSI,  and  get  working  algorithms  in  the  field  at  all  3  sites. 

3.  Statement  of  Work 

The  Statement  of  Work  for  Grant  N00244-07-1-009  is  given  in  italics  below. 

Primary  Task: 

Task  1.  Under  the  Primary  Task  funding  in  the  first  year  from  May  1  2007  through  April 
30  2008,  or  for  12  months  after  receipt  of  grant,  support  the  multi-site  research 
objectives  by  providing  the  necessary  test  hardware. 

Subtask  1.1.  Refurbish  an  existing  Whole  Sky  Imager  and  repair  and/or  replace 
failed  components  and  install  software/hardware  upgrades  necessary  to  ensure 
24/7/365  data  collection.  Begin  refurbishment  of  additional  parts/materials  to  allow 
for  low  to  moderate-cost  repairs  for  other  WSI  systems  in  the  field.  Test  and 
calibrate  the  refurbished  WSI  and  software  to  insure  suitability  for  deployment  to  a 
remote  field  site.  Install  the  refurbished  WSI  system  at  CONUS  field  site  #3.  During 
both  pre-deployment  and  post-deployment  intervals,  adapt  the  existing  software 
acquisition  algorithms  as  well  as  the  hardware  of  the  deployed  all  sky  cameras  to 
address  and  circumvent  minor  problems  that  arise  during  the  routine  operation  of  the 
cameras  in  the  field.  Problems  may  include  adapting  to  different  background  light 
levels,  system  sensitivity,  and  different  integration  times. 

Subtask  1.2.  Write  technical  memoranda  to  document  the  system  configuration,  test 
results,  and  related  information  that  may  be  needed  in  the  field.  Provide  a  report 
documenting  the  status  of  the  instrument  and  providing  an  overview  of  the  results  of 
the  test,  with  regard  to  its  anticipated  performance.  Report  results  at  occasional 
meetings  as  scheduled  by  the  DoD  contacts  at  Starfire  Optical  Range. 

Task  2.  Under  the  Primary  Task  funding  for  the  first  year  from  May  1  2007  through 
April  30  2008,  or  for  12  months  after  receipt  of  grant,  support  the  multi-site  research 
objectives  by  providing  analysis  of  the  acquired  data. 
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Subtask  2.1.  For  the  instrument  fielded  in  Item  1  and  a  second  WSI  which  is  already 
fielded,  perform  a  daytime  clear  sky  background  analysis  for  the  site,  lens,  detector 
and  system  combination,  update  the  cloud  algorithm  input  files  to  identify  cloud 
obscuration  and  create  cloud  decision  images.  Deliver  the  appropriate  input  files  to 
the  government  for  installation  in  the  field  to  enable  near-real  time  creation  of  cloud 
decision  images.  Evaluate  the  night  data,  and  extract  the  star  signals,  clear  sky 
background,  and  cloud  radiance  fields  as  necessary  to  update  the  algorithm  input 
files  for  creating  cloud  decision  images  at  night.  Deliver  the  input  files  to  the 
government  for  installation  in  the  field.  During  both  pre-deployment  and  post- 
deployment  intervals,  adapt  the  existing  analysis  and  support  software  to  address  and 
circumvent  minor  problems  that  arise  during  the  routine  operation  of  the  cameras  in 
the  field.  Problems  include  adapting  to  different  haze  levels,  and  issues  such  as 
instrument  sensitivity  that  may  impact  the  algorithms.  Evaluate  the  algorithm  results, 
and  assess  whether  they  should  be  improved,  and  how  they  might  be  improved  if 
necessary. 

Subtask  2.2.  Write  technical  memoranda  to  document  the  processing,  analysis  and 
results.  Provide  a  report  documenting  an  overview  of  the  status  of  the  instrument 
algorithms  and  the  results  of  analysis.  Report  results  at  occasional  meetings  as 
scheduled  by  the  DoD  contacts  at  Starfire  Optical  Range. 

The  Optional  Tasks  were  not  funded,  and  therefore  we  have  not  included  them  here. 

4.  Funding  Increments 

The  funding  for  the  primary  tasks  was  received  28  June  2007.  The  options  were  not 
funded.  It  should  also  be  noted  that  we  received  a  no-cost  extension  through  3 1  October 
2008  on  14  April  2008. 

5.  Major  Deliveries  and  Documentation 

This  section  outlines  the  major  deliveries  associated  with  each  task.  The  details  of  the 
hardware  and  software  deliveries  and  documentation  memo  numbers  are  discussed  in 
Sections  6  and  7.  The  details  of  the  algorithm  developments  and  data  analysis  and 
documentation  memo  numbers  are  discussed  in  Sections  8  and  9. 

Task  1:  This  task  is  directed  primarily  toward  the  refurbishment  a  WSI,  and  its 
deployment  at  Site  3.  It  also  includes  modification  of  hardware  and  software  as 
necessary,  and  calibration  of  the  system.  This  task  also  allows  for  refurbishment  of 
additional  systems.  Task  1.2  requires  documentation  of  this  work,  both  in  memos  and  in 
a  report. 

The  refurbishment  of  the  unit  for  Site  3  was  delayed  due  to  camera  repairs  at  the 
manufacturer  that  took  quite  some  time,  however  the  refurbishment  was  completed,  and 
the  system  deployed  on  April  9  2008.  In  addition  to  the  standard  refurbishments,  the 
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system  was  modified  by  adding  a  ring  shade,  as  will  be  discussed  in  Section  6. 
Calibrations  were  acquired  and  analyzed.  Also,  the  unit  at  Site  5  was  hit  by  lightning  in 
July  2007,  so  we  refurbished  it,  and  redeployed  it  January  16  2008.  We  continued 
monitoring  of  the  Site  2  instrument  and  data,  and  also  repaired  the  instrument  at  the  SOR, 
although  it  required  further  repairs  a  few  months  later.  We  prepared  a  spare  WSI  unit, 
Unit  9. 

Numerous  memos  were  provided  to  the  sponsor  or  to  SOR,  and  are  listed  in  Section  12.1. 
All  of  the  memos  written  between  July  2007  and  February  2008  were  written  with 
funding  from  this  grant,  and  the  later  memos  were  in  many  cases  at  least  partially  funded 
by  this  grant,  or  they  document  work  done  under  this  grant.  These  memos  document  the 
configuration  of  the  hardware,  the  software,  the  use  of  the  instrument,  the  calibration  tests 
and  their  results,  and  so  on.  The  requirement  for  a  report  documenting  the  hardware  is 
met  by  this  report,  Technical  Note  274.  The  memos  will  be  referenced  in  the  sections 
discussing  the  hardware. 

Task  2:  This  task  is  directed  primarily  toward  providing  algorithms  running  in  real  time 
at  Sites  3  and  5.  It  requires  extracting  the  field  data  in  able  to  determine  the  inputs  for 
both  day  and  night  algorithms.  Task  2.2  requires  documentation  of  this  work,  both  in 
memos  and  in  a  report. 

For  the  day  algorithm,  we  analyzed  Site  5  January  -  May  08  data.  We  found  that  in  order 
to  get  good  results,  we  had  to  upgrade  the  day  algorithm  in  order  to  include  an  adaptive 
feature  that  adjusted  for  haze  amount.  This  was  completed  and  integrated  into  the  field 
code  as  well  as  the  in-house  interactive  code.  The  inputs  completed  and  delivered  to  the 
sponsors,  who  installed  it  August  26  2008.  The  day  algorithm  was  further  updated  and 
installed  on  October  14  2008.  The  Site  3  day  algorithm  extraction  and  setup  was  also 
completed,  and  fielded  October  14  2008.  We  also  processed  significant  amounts  of  Site 
5  and  Site  3  archival  data,  and  delivered  it  to  SOR  on  October  23  2008. 

For  the  night  algorithm,  we  felt  that  we  needed  to  upgrade  the  algorithm  to  handle 
moonlight  well.  This  took  a  good  bit  of  analysis  and  programming,  but  the  job  was 
completed  using  IDL  code  and  then  converted  to  C  and  integrated  into  the  field  code. 
The  moonlight  algorithm  works  well.  The  Site  5  extraction  was  completed  and  the  code 
was  installed  in  the  field  August  26  2008,  and  further  upgraded  and  reinstalled  on 
September  23  2008.  The  Site  3  inputs  were  extracted,  and  installed  in  the  field  October 
23  2008.  Significant  amounts  of  Site  3  and  5  archival  data  were  processed,  and  delivered 
to  the  SOR  on  October  23  2008.  In  addition,  we  completed  an  analysis  of  optical  fade  at 
night,  and  presented  the  results  at  the  May  2008  meeting.  The  archival  processing  and 
optical  fade  work  were  primarily  funded  by  the  follow-on  contract  however. 

Numerous  memos  were  provided  to  the  sponsor  or  to  SOR,  and  are  listed  in  Section  12.1. 
All  of  the  memos  written  between  July  2007  and  February  2008  were  written  with 
funding  from  this  grant,  and  the  later  memos  were  in  many  cases  at  least  partially  funded 
by  this  grant,  or  they  document  work  done  under  this  grant.  These  memos  document  the 
format  and  use  of  the  archival  data,  and  the  use  of  the  real  time  data.  They  also  include 
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documentation  of  the  new  algorithms,  and  the  extracted  algorithm  inputs.  At  the  request 
of  SOR,  many  of  these  algorithm  input  memos  were  not  written  until  later  at  the  request 
of  SOR,  but  they  have  now  been  completed,  and  are  referenced  below.  The  requirement 
for  a  report  documenting  the  hardware  is  met  by  this  report,  Technical  Note  274.  The 
memos  will  be  referenced  in  the  sections  discussing  the  hardware. 

It  should  be  noted  that  both  tasks  also  require  attending  meetings  and  presenting  results. 
The  primary  author  attended  meetings  on  Nov  29  2007,  Jan  29  2008,  Jan  29  2008,  Feb  28 
2008,  Mar  20  2008,  May  6  2008,  May  29  2008,  Sep  1 1  2008,  and  Oct  23  2008.  She  had 
prepared  a  talk  for  the  Oct  23  2007  meeting,  but  a  major  wildfire  near  home  prevented 
participation.  A  total  of  1 1  talks  were  prepared  by  the  group  and  presented  at  these 
meetings.  An  overview  of  the  topics  of  each  talk  is  provided  in  Section  12.2. 

Although  this  Section  5  discussion  documents  some  of  the  more  significant  deliveries, 
there  were  numerous  additional  deliveries,  which  are  documented  in  the  following 
sections. 

We  believe  we  have  completed  all  contract  requirements  at  this  time. 

This  work  is  discussed  more  fully  in  the  following  Sections  6-9. 

6.  WSI  Refurbishment,  Deployment,  and  Support 

This  section  discusses  the  progress  with  WSI  refurbishment  and  deployment,  QC  and 
support  of  the  fielded  instruments  under  this  Grant. 

6.1.  WSI  Refurbishment  under  the  previous  contracts 

As  discussed  in  Technical  Notes  272  and  273,  some  older  WSI  systems  became  available 
early  in  2005,  and  the  decision  was  made  to  refurbish  these  systems  for  use  at  three  new 
SOR  sites  (Sites  2,  3,  and  5)  for  this  program.  These  older  instruments  were  originally 
built  for  the  Department  of  Energy  (DOE)  and  used  for  many  years  at  a  variety  of  sites. 
Following  refurbishment  of  the  instruments,  WSI  Unit  7  was  deployed  to  Site  2  in  May 
2006,  and  Unit  4  was  deployed  to  Site  5  in  January  2007.  As  discussed  in  Technical 
Notes  272  and  273,  this  work  included  a  lot  of  upgrade  of  capabilities,  as  well  as  repair 
and  preventative  maintenance.  In  addition,  quite  a  bit  of  work  was  completed  on  Unit  8 
under  the  previous  contract. 

The  configurations  of  the  Unit  7  sensor  and  the  controller  upon  completion  of  the 
refurbishment  are  shown  in  Figure  1.  The  left  side  shows  the  sensor  unit,  with  its 
environmental  housing  and  the  solar/lunar  occultor.  This  unit  has  successfully  operated 
for  many  years  in  the  Arctic,  and  similar  units  have  operated  in  other  locations,  including 
the  tropics  and  the  desert.  The  right  side  shows  the  controller  unit,  as  reconfigured  for 
the  SOR  project.  The  other  units  are  quite  similar,  except  for  the  size  of  the  occultor 
shade,  which  depends  on  the  latitude  of  the  site.  Also,  Units  7  and  8  have  glass  domes, 
and  Unit  4  has  an  acrylic  dome. 
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Fig.  1.  Sensor  and  Controller  Configuration  for  Unit  7 


These  systems  were  built  in  the  mid-to-late  1990’s,  and  as  a  result  they  needed  a  fair 
amount  of  refurbishment.  Typical  refurbishment  tasks  include  disassembly  and  cleaning, 
replacing  worn  components  such  as  the  coolant  tubing,  replacing  any  failed  components 
such  as  arc  drive  motors,  and  getting  the  cameras  tested  and  purged  at  Photometries.  In 
many  cases,  these  components  were  still  operational,  but  it  was  sensible  to  replace  them 
prior  to  redeployment. 

6.2.  Site  3  Deployment 

Under  this  contract,  we  continued  with  the  refurbishment  of  Unit  8.  The  biggest  problem 
with  Unit  8  was  that  the  camera  output  was  intennittent.  We  sent  it  to  Photometries  in 
September  2007,  however  they  no  longer  had  the  software  or  knowledge  to  operate  the 
camera.  One  of  our  team  members  went  out  to  Photometries  and  spent  several  days 
working  with  them,  and  they  were  able  to  repair  the  camera.  The  camera  repair  was 
completed  and  the  camera  was  returned  on  27  November  07.  The  Photometries  trip  and 
repair  procedures  are  documented  in  Memo  AV07-052t. 

Following  repair  of  the  camera,  we  completed  the  rest  of  the  refurbishment,  test,  and 
calibration,  and  prepared  the  documentation  that  is  required  for  fielding.  The  instrument 
was  fielded  during  8-10  April  2008.  The  documentation  memos  are  AV08-004t,  which 
lists  the  start-up  procedures,  AV08-005t,  which  provides  a  check  list,  AV08-006t,  which 
provides  an  overview  of  the  control  computer  operations,  AV08-007t,  which  provides  an 
overview  of  the  processing  computer  operations,  AV08-008t,  which  provides  the  set-up 
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instructions,  AV08-009t,  which  discusses  the  system  focus  and  calibration,  and  AV08- 
OlOt,  which  provides  the  system  overview  and  parts  list.  Memo  AV08-013t  provides  the 
installation  report. 

The  site  installation  is  shown  in  Fig.  2.  We  had  been  told  that  there  were  a  lot  of  lights 
around  the  horizon,  so  to  address  this  issue  we  designed  a  small  ring  shade  to  place 
around  the  dome.  This  ring  shade  can  perhaps  be  seen  in  Fig.  2,  and  is  documented  in 
Memo  AV09-054t.  We  tested  the  imagery  both  with  and  without  the  ring  shade.  As 
shown  in  Fig.  3,  the  results  without  shading  were  quite  poor,  but  with  the  ring  shade 
installed,  the  results  were  excellent.  The  ring  shade  blocks  the  top  of  the  dome  from  any 
lights  that  are  5  degrees  above  the  horizon  or  less.  This  results  in  blocking  12  degrees 
above  the  horizon  in  the  field  of  view,  since  the  base  of  the  dome  is  lower  than  the  top  of 
the  dome. 


Fig.  2.  Site  3  installation,  including  WSI,  weather  instruments,  and  shed  with  computers 


When  the  site  started  getting  warm,  we  had  problems  with  temperature  inside  the 
housing.  On  3  -  5  June  08,  we  replaced  the  Teca  cooler  with  another  Teca  cooler,  as 
documented  in  Memo  AV08-020t.  This  did  not  turn  out  to  solve  the  problem,  indicating 
that  the  problem  was  that  the  Teca  model  of  cooler  is  not  sufficient  for  this  site.  As  a 
result,  on  28  -  30  July  08,  we  made  a  second  trip  and  completely  replaced  the 
environmental  housing  for  another  we  had  refurbished  at  MPL  that  included  a  McLean 
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cooler.  The  McLean  cooler  has  twice  the  BTU  of  the  Teca.  This  trip  is  documented  in 
Memo  AV08-030t.  This  solved  the  cooler  problem. 


Fig.  3.  Raw  image  at  night  with  no  stray  light  blocking  and  with  stray  light  blocking 


There  were  also  occasional  problems  with  the  occultor  during  this  time.  The  brake  power 
supply  was  readjusted  in  the  field  by  the  SOR  team  in  May  08.  In  addition,  the  occultor 
stopped  working  on  August  10,  and  then  recovered  by  itself  on  September  2,  and 
continued  working  well  throughout  the  period  of  the  contract.  After  the  end  of  this 
contract,  some  problems  occurred  again  and  the  occultor  was  further  adjusted. 

In  spite  of  the  initial  start-up  problems,  most  of  the  data  for  this  site  were  good.  The  site 
statistics  chart  is  shown  in  Fig.  4.  In  spite  of  the  problems,  the  system  acquired  85% 
good  data  during  the  first  few  months,  and  98%  good  data  in  the  last  two  months  of  the 
contract  period. 

6.3.  Site  2  Support 

Site  2  had  been  installed  in  May  2006,  and  as  reported  in  Technical  Note  273,  it  worked 
reasonably  well  in  spite  of  being  an  old  instrument  and  the  first  deployment  of  a 
refurbished  system.  We  acquired  approximately  85%  good  data  during  the  period  1  May 
to  10  December  2006.  There  was  an  occultor  failure  on  11  December  2006. 
Unfortunately  we  had  insufficient  funding  to  do  this  repair,  however  the  SOR  team  was 
able  to  repair  it  with  our  support  in  late  February  07,  as  documented  in  Memo  AV07- 
038t.  During  the  period  28  Feb  through  30  Sep  2007,  the  system  acquired  98%  good 
data,  as  shown  in  Technical  Note  273. 

The  system  continued  to  run  well  until  February  2008,  when  the  shutter  became 
intennittent.  It  continued  to  acquire  mostly  good  data,  and  the  repair  was  made  during  13 
-  16  April  2008,  as  documented  in  Memo  AV08-014t.  This  repair,  and  additional  repairs 
made  later,  were  funded  by  the  follow-on  contract,  however  I  will  document  the  repairs 
up  until  the  end  of  Oct  3 1  08,  which  was  the  end  of  this  contract  period.  On  10  May,  the 
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control  computer  failed.  We  sent  a  replacement  computer,  however  the  site  personnel 
were  unable  to  get  it  running.  We  ended  up  having  to  replace  the  processing  computer  as 
well,  and  this  was  completed  3-5  June  2008,  as  documented  in  Memo  AV08-024t. 


UNCLASSIFIED 

Site  3  Operational  Statistics 


UNCLASSIFIED 


Fig.  4.  Site  statistics  for  Site  3  during  the  contract  period 

Also,  the  cooler  was  destroyed  1  August  2008,  as  a  result  of  a  site  power  surge,  and  the 
system  had  to  be  turned  off.  It  was  repaired  11-14  November  2008,  as  documented  in 
Memo  AV08-046t. 

Thus,  during  the  period  of  the  contract  from  10  July  2007  through  31  October  2008,  this 
system  ran  during  10  July  2007  through  10  May  2008  and  3  June  through  31  July  2008. 
The  status  chart  for  these  periods  is  shown  in  Fig.  5.  Although  the  statistics  are  not 
optimal,  they  are  reasonable  considering  that  there  was  not  adequate  funding  to  repair  this 
system  during  much  of  this  contract  period.  During  the  first  period  the  system  acquired 
88%  good  data,  and  during  the  second,  it  acquired  80%  good  data. 

6.4.  Site  5  Support 

Site  5  was  deployed  31  January  2007,  as  documented  in  Technical  Note  273.  It  worked 
quite  well  until  it  was  hit  by  lighting  on  23  July  2007.  As  shown  in  Technical  Note  273, 
during  this  interval  it  acquired  91%  good  data,  and  most  of  the  lost  data  were  due  to 
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facility  failure,  rather  than  any  problems  with  the  WSI.  During  this  interval,  there  was 
some  routine  maintenance,  including  replacing  the  external  drive,  performed  on  27  -  28 
June  2007,  and  documented  in  Memo  AV07-039t. 


UNCLASSIFIED 

Site  2  Operational  Statistics 


10  Jul.  2007  -  9  May  2008  1  Jun.  -  31  Jul.  2008 


Jul.  1,  2007  -  May  9, 2008:  88.2%  good  data  of  which  <  1%  flux  Ctrl  not 
optimal.  Intermittent  shutter  problems  beginning  Aug.  15, 2007. 

Problems  with  shutter  become  more  regular  beginning  in  Feb.  2008. 

□ 

U  of  actual  image  sets  grabbed 

System  repaired  Apr.  14-16  2008. 

a 

#  of  image  sets  lost  during  full  moon 

□ 

#  of  image  sets  lost  due  to  timing  issues 

Jun.  1  -  Jul.  31  2008:  80.3%  good  data  of  which  <  1%  flux  Ctrl  not 
optimal.  System  was  down  June  1-3  due  to  control  computer 

□ 

U  of  image  sets  lost  due  to  facility  failure 

issues. 

□ 

U  of  image  sets  lost  due  to  data  issues 

□ 

U  of  image  sets  where  flux  control  is  not 
optimal 

UNCLASSIFIED 


Fig.  5.  Site  statistics  for  Site  2  during  the  contract  period 

Although  this  grant  was  not  intended  to  fund  major  repairs,  repair  of  the  lightning  strike 
was  a  major  priority  to  the  program,  and  as  a  result  we  requested  permission  to  work  on 
this  repair.  In  August,  the  SOR  team  went  to  the  site,  and  returned  the  computer  and  both 
ACP’s  (Sensor  Accessory  Control  Panel  and  Occultor  Accessory  Control  Panel,  used  to 
electronically  control  the  system).  These  were  repaired  at  MPL.  A  trip  was  made  by 
MPL  during  October  18  -  19  2007  to  reinstall  these  components,  as  documented  in 
Memo  AV07-050t.  On  this  trip,  we  tested  all  the  components  in  detail,  and  found  that 
the  brake  power  supply  and  the  camera  were  also  dead.  On  return  of  these  components  to 
MPL,  we  did  further  tests  of  the  camera,  and  then  returned  it  to  Photometries  for  repair, 
as  documented  in  Memo  AV07-052t.  On  return,  the  camera  housing  was  reassembled. 
We  had  also  had  a  problem  in  2007,  in  that  the  daytime  images  were  very  slightly  out  of 
focus.  This  was  fixed,  and  the  system  recalibrated,  as  documented  in  Memo  AV08-00L. 
The  system  was  redeployed  during  15-17  January  2008,  as  documented  in  Memo 
AV08-002t. 
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The  system  operated  well  from  16  January  2008  through  22  May  2008,  when  the  occultor 
ACP  lost  power.  This  was  repaired  during  30  June  through  2  July  2008,  as  documented 
in  Memo  AV08-026t,  and  a  ring  shade  was  added  at  that  time.  After  that,  it  operated  well 
through  the  remainder  of  the  contract  period  ending  31  October.  The  status  chart  is 
shown  in  Fig.  6.  We  achieved  97%  good  data  during  16  Jan  -  22  May,  and  93%  good 
data  during  Jul  1  -  Oct  31.  Most  of  this  data  loss  was  due  to  the  system  accidentally 
being  turned  off  for  a  few  days. 


UNCLASSIFIED 

Site  5  Operational  Statistics 


16  Jan. -22  May  2008  1  Jul  -  31  Oct.  2008 


Jan.  16  -  May  22, 2008:  97,1%  good  data  of  which  approx.  7%  flux  Ctrl 
not  optimal.  Occultor  ACP  lost  power  on  May  22. 

□ 

#  of  actual  image  sets  grabbed 

Jul.  1  -  Oct.  31 , 2008:  93.5%  good  data  of  which  approx.  9%  flux  Ctrl 

a 

#  of  image  sets  lost  during  full  moon 

not  optimal.  Sensor  ACP  accidently  powered  off  by  SOR  personnel 

□ 

#  of  image  sets  lost  due  to  timing  issues 

July  11  -16. 

□ 

#  of  image  sets  lost  due  to  facility  failure 

□ 

U  of  image  sets  lost  due  to  data  issues 

CD 

U  of  image  sets  where  flux  control  is  not 
optimal 

UNCLASSIFIED 


Fig.  6.  Site  statistics  for  Site  5  during  the  contract  period 

6.5.  SOR  Site  and  Virginia  Site 

We  were  not  funded  under  this  contract  to  support  either  of  these  sites.  They  had  been 
down  for  some  time  due  to  lack  of  funding.  However,  because  a  team  member  had  to  go 
to  the  SOR  site  to  give  presentations,  we  were  able  to  provide  repairs  at  no  cost  to  the 
program.  We  brought  the  SOR  (Site  7)  WSI  Unit  #12  on-line  on  27  September  2007,  as 
documented  in  Memo  AV07-047t.  We  also  did  preventative  maintenance  on  29 
November  2007,  as  documented  in  Memo  AV07-057t.  Unfortunately,  the  system  would 
hang  about  once  a  day  when  acquiring  an  image.  The  SOR  team  tried  to  diagnose  it  in 
December  07,  and  the  system  failed. 
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When  the  follow-on  contract  came  through  in  February  2008,  we  were  funded  to  support 
the  SOR  site,  although  this  was  lower  priority  than  supporting  the  other  sites.  The  SOR 
site  was  brought  on-line  during  16  -  18  June  08  with  funding  from  the  follow-on 
contract,  as  documented  in  Memo  AV08-024t.  It  continued  to  work  well  through  the  end 
of  the  contract  period,  although  we  lost  quite  a  bit  of  data  because  of  power  issues  at  the 
site.  The  status  chart  is  shown  in  Fig.  7. 


UNCLASSIFIED 

Site  7  Operational  Statistics 

17  Jun.  -31  Oct.  2008 


Fig.  7.  Site  statistics  for  Site  7  during  the  contract  period 

6.6.  Hardware  Summary 

Under  this  contract,  Unit  8  was  deployed  to  Site  3,  the  last  of  the  main  sites.  Site  2  was 
kept  in  repair,  and  the  Site  5  instrument  received  extensive  repairs  following  a  lightning 
strike,  although  this  work  was  partly  funded  under  the  follow-on  contract.  Quite  a  bit  of 
additional  hardware  work  took  place  during  the  interval  of  the  contract,  but  starting  Feb 
2008  was  mostly  funded  under  the  follow-on  contract. 

7.  Software  Upgrades  and  Site  Data  Processing 

This  section  discusses  the  upgrades  to  the  field  software  and  the  archival  and  processing 
of  data.  Although  this  section  discusses  the  new  algorithm  installation,  the  actual 
algorithm  upgrades  are  discussed  in  Sections  8  and  9. 
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7.1.  Field  Software  Status 


There  are  two  primary  programs  used  in  the  field.  RunWSI  provides  the  system  control, 
and  ProcWSI  provides  the  result  of  the  real  time  algorithms  that  are  then  sent  to  SOR  via 
protected  ftp.  The  use  of  these  programs  is  documented  in  Memos  AV08-006t  and  -007t, 
and  the  quick  shut-down  is  documented  in  Memo  AV08-004t.  In  addition,  an  in-house 
version  of  the  cloud  algorithm,  AutoProc,  is  used  for  testing  the  day  algorithm  as  it 
develops,  and  various  IDL  programs  are  used  for  testing  the  night  algorithm  as  it 
develops.  Numerous  test  and  data  control  programs  are  also  present  both  in  the  field  and 
in  the  home  analysis  facility. 

RunWSI  is  designed  to  control  the  instruments.  It  determines  the  time  and  location; 
calculates  the  desired  occultor  position,  filter  selection  and  exposure;  moves  the  occultor 
and  filter  into  position;  acquires  the  images  at  the  proper  exposure  and  in  the  desired 
filters;  performs  initial  QC;  write  status  information  to  a  header  as  well  as  QC  files;  and 
then  sends  the  images  to  the  processing  computer.  ProcWSI  runs  on  the  processing 
computer.  Its  primary  purpose  is  to  accept  the  raw  imagery,  and  compute  the  cloud 
decision  using  the  algorithms  documented  in  Sections  8  and  9.  It  performs  additional  QC 
checks,  and  stores  these  results  as  well  as  the  processing  parameters  in  the  headers.  It 
then  archives  the  data  on  an  external  drive,  and  also  ships  the  data  via  protected  ftp  to  the 
SOR  site. 

The  field  software  was  completely  upgraded  under  the  previous  two  contracts,  as 
documented  in  Technical  Notes  272  and  273.  As  a  result,  very  few  upgrades  were 
needed  under  this  contract.  The  RunWSI  program  did  not  require  any  upgrades  during 
this  period.  The  specific  version  of  each  system  and  its  history  of  fielded  software  is 
documented  in  Memo  AV09-005t,  which  includes  both  the  RunWSI  and  the  ProcWSI 
history. 

The  programs  write  “headers”  to  the  images  in  order  to  document  the  acquisition  and 
processing  settings,  and  also  to  indicate  the  results  of  automated  quality  control  checks. 
The  headers  created  by  RunWSI  are  documented  in  Memo  AV07-054t.  We  added  new 
headers  to  the  ProcWSI  program.  It  was  installed  in  the  field  in  Oct  08  at  Sites  3  and  5, 
as  documented  in  Memo  AV09-005t.  The  new  headers  are  documented  in  Memo  AV08- 
037t.  The  program  updates  are  documented  in  Memo  AV08-05  It. 

A  new  program,  QCStat,  was  written  to  expedite  the  quality  assessment,  and  this  is 
documented  in  Memo  AV07-05  It.  This  program  made  the  assessment  of  field  data  much 
easier,  and  enabled  us  to  start  making  monthly  assessments  of  the  data. 

The  most  significant  upgrade  to  the  field  software  was  the  addition  of  the  new  adaptive 
day  cloud  algorithm  and  the  new  moonlight  algorithm  for  the  night.  These  upgrades 
were  fielded  at  Sites  3  and  5  during  the  August  -  October  2008  period,  as  documented  in 
Section  5.  The  algorithm  upgrades  are  discussed  in  detail  in  Sections  8  and  9,  and 
documented  in  Memos  AV09-040t,  -04  It,  and  -046t.  The  details  of  the  software  upgrade 
for  ProcWSI  are  documented  in  Memo  AV08-051t.  This  upgrade  also  included  addition 
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of  an  overall  quality  indicator.  It  is  important  that  users  of  the  data  base  be  aware  of  this 
indicator,  as  will  be  discussed  in  Section  7.2. 

There  are  many  support  programs  used  in  handling  these  large  volumes  of  data,  and  in 
extracting  the  field  data  and  setting  up  the  algorithm  inputs.  Many  of  these  programs 
were  updated  significantly  during  this  period,  as  documented  in  Memos  AV08-029t,  - 
03  It,  -032t,  -033t,  -034t,  -035t,  -036t,  and  -039t.  However  this  work  was  funded  under 
the  follow-on  contract,  and  thus  is  not  discussed  here. 

7.2.  Data  Archival  and  Processing 

The  instruments  deployed  to  Sites  2,  3  and  5  include  external  hard  drives  (originally  300 
GB,  now  500  GB).  The  primary  data  archival  occurs  at  SOR,  and  data  are  ftp’d  directly 
from  the  processing  computer  to  the  SOR  site,  where  they  are  archived.  The  external 
drives  are  a  redundant  backup  that  provide  a  second  archive,  in  case  of  server  crashes  at 
the  SOR  site. 

The  300  GB  drives  must  be  changed  approximately  every  6  months,  and  the  500  GB 
drives  every  9  months.  The  procedure  for  changing  the  drive  is  documented  in  Memo 
AV07-017t. 

As  discussed  in  Section  5,  our  primary  goal,  in  addition  to  data  acquisition,  was  to  get  the 
real  time  algorithms  in  the  field  for  Site  3  and  5.  In  addition,  we  went  ahead  and 
processed  archival  data  as  time  permitted,  since  this  was  a  high  priority  at  SOR.  The  data 
that  were  acquired  and  processed,  as  well  as  the  delivery  of  the  real  time  algorithms,  is 
documented  in  Table  1.  Some  of  the  archival  processing  and  algorithm  development 
after  1  Feb  08  was  funded  by  the  previous  contract. 

Table  1  lists  the  good  data  that  had  been  acquired  as  of  31  Oct  08,  in  columns  1-4.  In 
column  3,  the  entry  “Alg  Fielded”  or  “Alg  Updated”  indicate  that  we  put  the  real  time 
algorithm  into  the  field.  For  those  archival  data  that  have  been  processed,  as  well  as  the 
real  time  algorithms  that  have  been  installed,  we  indicate  the  date  of  the  delivery  in 
Column  4,  the  memo  which  documents  what  was  delivered  and  how  to  use  it  in  Column 
5,  and  the  memo  which  documents  the  determination  of  the  inputs  and  the  results  in 
Column  6. 

In  the  case  of  Site  3  night,  we  had  not  yet  processed  any  of  the  archival  data  at  the  end  of 
October  2008,  however  we  had  analyzed  it  and  set  up  the  algorithm  inputs.  The  real  time 
algorithm  inputs  were  based  on  this  analysis,  and  for  this  reason  we  have  included  the 
processing  memos  for  these  data  in  the  above  table.  Because  the  real  time  algorithm  was 
updated  on  12/05/08,  we  only  documented  the  final  delivery  in  Table  1.  The  initial 
algorithm  results  were  nearly  as  good,  however  we  will  be  reprocessing  the  data  between 
10/23/08  and  12/05/08. 
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Table  1 

Field  Data  Archival  and  Processing  Status  thru  3 1  Oct  08 


Site 

Day/Night 

Period 

Date 

Delivered 

Memo  on  Delivery 

Memo  on 
Processing 

2 

Day 

5/2/06-12/11/06 

12/12/06-2/26/07 

No  data 

2/27/07  -  12/31/07 

1/1/08-7/31/08 

8/1/08-10/31/08 

No  data 

Night 

5/2/06-12/11/06 

12/12/06-2/26/07 

No  data 

2/27/07  -  12/31/07 

1/1/08-7/31/08 

8/1/08-10/31/08 

No  data 

3 

Day 

4/9/08  -  8/4/08 

10/23/08 

AV08-0381 

AV09-0081 

8/5/08  -  10/14/08 

Alg  Fielded 

10/14/08 

AV09-024t 

AV09-0081 

Night 

4/9/08  -  7/31/08 

AV09-048t 

8/1/08  -  9/1/08 

AV09-049t 

9/2/08  -  10/28/08 

AV09-050t 

Alg  Fielded 

10/23/08 

AV09-050t 

10/29/08  -  12/05/08 

Alg  Updated 

12/05/08 

AV09-024t 

AV09-050t 

5 

Day 

2/1/07  -  7/13/07 

7/14/07-1/15/08 

No  data 

1/16/08  -  5/20/08 

10/23/08 

AV08-0381 

AV09-042t 

5/21/08  -  5/22/08 

AV09-042t 

5/23/08  -  6/30/08 

No  data 

7/1/08  -  10/31/08 

AV09-043t 

Alg  Fielded 

8/26/08 

AV09-043t 

Alg  Updated 

10/14/08 

AV09-024t 

AV09-043t 

Night 

2/1/07  -  7/12/07 

10/23/08 

AV08-0381 

AV09-05U 

7/14/07-1/15/08 

No  data 

1/17/08  -  2/20/08 

AV09-0521 

2/21/08  -  6/30/08 

10/23/08 

AV08-038t 

AV09-0521 

7/1/08  -  8/4/08 

10/23/08 

AV08-0381 

AV09-0531 

8/1/08  -  9/30/08 

Alg  Fielded 

8/26/08 

AV09-053t 

Alg  Updated 

9/23/08 

AV09-024t 

AV09-053t 

Memo  AV08-038t  documents  the  specific  data  delivered,  and  important  infonnation  such 
as  the  geometric  calibration  equations  to  use  with  the  data.  In  addition.  Memo  AV08- 
037t  provides  an  overview  of  the  archival  fonnat  and  its  use.  Memo  AV09-024t 
documents  the  real  time  algorithms  that  were  delivered  to  the  field.  The  memos  in 
Column  6  document  the  setup  of  the  algorithm  inputs  for  the  specific  data  set,  as  well  as 
the  results.  Many  of  the  memos  in  Column  6  were  written  much  after  the  contract 
ended,  at  the  request  of  SOR.  Because  we  feel  that  Memos  AV08-037t  and  -038t  and 
AV09-024t  are  important  to  the  data  base  user,  we  have  repeated  them  in  Appendices  1  - 
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3.  It  should  also  be  noted  that  we  have  recently  found  an  error  in  the  night  algorithm 
header  where  the  cloud  fraction  numbers  are  reported.  We  will  be  correcting  all  these 
data  and  redelivering  them.  The  algorithm  results  in  the  pixels  are  valid,  but  the 
summary  cloud  fraction  numbers  in  the  night  headers  are  not. 

At  the  time  of  writing  this  report,  far  more  data  has  been  acquired  and  processed.  This 
will  be  discussed  in  the  report  for  the  follow-on  contract. 

The  results  of  the  processing  are  discussed  in  Sections  8  and  9,  which  discuss  the  cloud 
algorithms  and  data  processing  results. 

8.  Day  Algorithm  Upgrade  and  Analysis 

During  the  previous  contract,  as  documented  in  Technical  Note  273,  we  had  processed 
and  evaluated  data  from  Site  2,  and  set  up  the  real  time  algorithm  running  at  Site  2.  This 
algorithm  used  NIR/blue  data,  because  Site  2  was  quite  hazy,  and  the  NIR  results  were 
better.  Also  during  that  period,  we  had  updated  the  algorithm  to  allow  the  clear  sky 
background  reference  value  to  change  as  a  function  of  Solar  Zenith  Angle  (SZA).  The 
results  were  reasonably  good,  but  we  felt  that  it  would  be  very  helpful  to  complete  the 
adaptive  algorithm,  which  would  adjust  for  variable  haze  amount  and  its  impact  on  the 
imagery.  Examples  and  more  detailed  discussion  of  these  previous  results  are  included  in 
Technical  Note  273,  Section  8. 

When  we  received  funding  to  begin  work  on  the  day  algorithm,  we  began  with  Site  5, 
which  was  the  next  site  that  had  been  fielded.  During  our  time  working  on  Grant 
N00244-07- 1-009,  we  were  able  to  complete  the  adaptive  algorithm.  We  also  extracted 
the  algorithm  inputs  for  Sites  5  and  then  3,  got  the  updated  algorithm  running  in  real  time 
at  both  sites,  and  processed  much  of  the  daytime  archival  data  for  these  two  sites. 
Processing  of  the  remaining  archival  data  as  well  as  getting  the  adaptive  version  of  the 
algorithm  running  in  the  field  at  Site  2,  were  completed  under  the  follow-on  contract. 
The  adaptive  algorithm  is  discussed  in  Section  8.1.  The  results  for  Sites  5  and  3  are 
discussed  in  Sections  8.2  and  8.3. 

8.1.  The  Day  Adaptive  Algorithm  that  adjusts  for  variations  in  Haze  Amount 

The  adaptive  algorithm  is  an  update  to  the  D/N  WSI’s  Daytime  cloud  decision  algorithm. 
It  is  designed  to  detect  cases  with  enhanced  haze,  and  adjust  the  algorithm  so  that  these 
cases  are  not  identified  as  thin  clouds.  Our  first  version  of  the  adaptive  algorithm  was 
programmed  for  the  Daylight  Visible/NIR  WSI  EO  System  7  that  was  built  for  the 
Germany’s  Deutsche  WetterDienst.  The  earlier  work  was  documented  in  Memo  AV04- 
012t  and  other  memos.  An  overview  of  the  version  of  the  adaptive  algorithm  written  for 
the  D/N  WSI  is  provided  in  Memo  AV09-040t,  and  summarized  here. 
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8.1.1.  Background 


The  adaptive  algorithm  had  been  adapted  and  re-programmed  for  the  D/N  WSI  earlier  in 
2007  or  2008,  but  it  did  not  work  well,  and  we  did  not  have  the  opportunity  at  that  time  to 
look  into  it  in  more  detail.  When  we  began  processing  the  Site  5  data,  however,  we 
found  that  the  haze  amount  was  quite  variable  for  this  site,  and  we  really  needed  to  debug 
the  adaptive  algorithm  and  get  it  working  well.  As  a  result,  we  went  ahead  and 
completed  this  work  during  July  through  September  2008,  and  used  it  for  the  Site  5  data 
analysis. 

The  Day  Thin  Cloud  algorithm  without  the  adaptive  haze  feature  uses  the  imagery  of  the 
sky  measured  with  blue  filters  and  either  red  or  NIR  filters.  The  initial  steps  in  the 
algorithm  are  essentially  calibration  corrections,  which  include  dark  correction,  non¬ 
uniformity  correction.  We  have  found  that  non-linearity  correction  is  not  necessary  with 
this  system,  because  the  corrections  are  so  small.  From  the  corrected  data,  either 
Red/blue  or  NIR/blue  ratios  are  computed  at  each  pixel.  These  can  be  thresholded  to 
provide  detection  of  opaque  clouds.  We  also  check  for  pixels  that  are  offscale  bright  or 
dark,  or  that  have  ratios  that  are  offscale. 

In  our  early  research,  we  found  that  this  fixed  threshold  worked  quite  well  for  opaque 
clouds  for  all  solar  zenith  angles,  and  all  conditions  we  have  encountered.  Originally,  we 
had  to  apply  a  zenith-angle-dependent  correction,  but  during  this  contract,  we  added  the 
non-uniformity  calibration  correction,  and  as  a  result  we  no  longer  have  to  apply  an 
additional  zenith-angle-dependent  correction.  Thus  the  correction  appears  to  be  due  to 
filter  and  lens  affects  that  are  corrected  with  the  uniformity  correction,  rather  than 
atmospheric  optical  effects. 

Thin  clouds  can  be  better  described  as  having  a  ratio  that’s  slightly  higher  than  the 
nonnal  clear  sky  ratio  (as  opposed  to  a  fixed  ratio  such  as  is  used  with  the  opaque 
clouds).  Accordingly,  techniques  were  developed  several  years  ago  to  characterize  the 
clear  sky  ratio  as  a  function  of  look  angle  and  solar  zenith  angle.  This  clear  sky 
background  ratio  is  extracted  from  cloud-free  images  acquired  at  the  site  in  the  field,  and 
stored  as  a  file  of  nonnalized  ratios  as  a  function  of  solar  zenith  angle  and  look  angle. 
When  field  data  are  processed,  a  clear  sky  background  ratio  image  is  generated  for  each 
field  image,  based  on  the  current  solar  zenith  angle. 

Once  the  clear  sky  background  image  has  been  generated,  those  pixels  in  the  current  ratio 
image  that  are  not  identified  as  opaque  or  offscale  are  compared  with  the  clear  sky  ratio. 
A  perturbation  ratio  is  computed  for  each  pixel.  This  perturbation  ratio  is  the  ratio  of  the 
current  pixel  red/blue  ratio  divided  by  the  clear  sky  background  library  ratio.  (The 
perturbation  ratio  shows  how  much  the  current  ratio  is  perturbed  or  changed  with  respect 
to  a  the  clear  sky  background  library  ratio  for  this  site,  time,  and  direction.)  If  the 
perturbation  ratio  exceeds  a  threshold  (typically  1.2  or  a  20%  change),  then  the  pixel  is 
identified  as  thin  cloud.  Also,  if  the  clear  sky  background  ratio  is  higher  than  the  opaque 
threshold,  the  pixel  will  be  identified  as  “indeterminate”,  meaning  that  the  clear  sky  ratio 
in  this  direction  for  this  solar  zenith  angle  is  anticipated  to  be  higher  than  the  ratio  for 


19 


opaque  clouds,  and  thus  we  can’t  identify  whether  there  are  clouds  there.  An  example 
will  be  shown  later. 

8.1.2.  Requirement  for  an  Adaptive  Algorithm 

In  general,  the  version  of  the  algorithm  described  in  Section  8.1.1  works  quite  well. 
Memos  AV06-018t  and  -020t  show  results  for  two  sites.  The  weakness  of  this  earlier 
version  of  the  algorithm  was  that  when  the  haze  amount  is  significantly  different  from 
that  of  the  cloud-free  cases  used  in  extracting  the  clear  background  library,  the  red/blue 
(and  NIR/blue)  ratio  is  higher,  and  the  sky  can  be  falsely  identified  as  thin  cloud.  Haze 
will  in  fact  attenuate  the  signal  of  anything  such  as  a  laser  going  through  the  atmosphere, 
just  as  thin  clouds  will,  but  it  attenuates  much  less  in  the  Short  Wave  IR  than  in  the 
visible  due  to  the  smaller  drop  sizes.  As  a  result,  we  want  to  be  able  to  distinguish  haze 
from  thin  cloud,  since  many  applications  are  in  other  non-visible  wavelengths. 

To  illustrate  the  problem,  Fig.  8  shows  the  raw  radiances  for  a  reasonably  clear  day  and  a 
fairly  hazy  day  from  the  same  site,  which  is  Site  5.  In  Fig.  8,  we  see  that  the  radiances 
for  both  the  blue  and  the  red  are  higher  with  haze  (these  images  were  windowed  to  the 
same  count  levels,  so  the  apparent  brightness  differences  are  real). 


Fig.  8.  Raw  Imagery,  windowed  with  the  same  count  levels,  for  a  clear  day  and  a  hazy  day  at  Site  5. 
Images  acquired  at  1700  GMT. 


Fig.  9  shows  the  Red/Blue  ratios  for  these  same  two  cases.  Note  that  the  ratio  is  higher  in 
the  hazier  case  (that  is,  the  sky  is  not  only  brighter,  but  also  whiter  when  it’s  hazy). 
Although  usually  the  change  with  haze  is  small,  in  this  case,  the  change  was  large  enough 
that  the  whole  sky  would  have  been  identified  as  thin  cloud. 
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Clear  Day  2/27/08 

Red/Blue  Ratio  Image 


Hazy  Day  4/27/08 


Ratios  are  also  higher  under  haze,  leading  to  a  false  result 

Fig.  9.  The  Red/Blue  ratio  image  for  the  same  cases  as  shown  in  Fig.  8.  Note  how  much  higher  the  ratio  is 
under  the  very  hazy  case. 

Fig.  10  shows  the  two  cases,  and  on  the  left  are  the  results  with  the  previous  algorithm, 
and  on  the  right  are  the  results  with  the  new  adaptive  algorithm.  In  Fig.  10,  yellow  is 
where  the  algorithm  has  called  it  “thin  cloud”,  and  blue  is  “no  cloud”.  Black  is  “no  data”. 
In  the  lower  right  image  there  is  a  grey  area  around  the  sun.  This  is  a  region  where  the 
algorithm  assigned  the  category  “indeterminate”,  as  discussed  in  Section  8.1.1. 

8.1.3.  Concepts  Behind  the  Adaptive  Algorithm 

When  we  first  evaluated  the  behavior  of  clear  sky  background  several  years  ago,  we 
found  that  the  relative  variation  in  ratio  as  a  function  of  look  angle  and  solar  zenith  angle, 
appears  to  be  nearly  independent  of  haze  amount.  Only  the  magnitude  changes 
significantly.  The  exceptions  are  right  near  the  aureole,  where  the  drop  size  distribution 
will  drive  the  shape  of  the  ratio,  and  also  right  near  the  horizon,  where  we  are  probably 
more  affected  by  the  height  of  the  haze  boundary  layer.  This  determination  was  made 
using  both  model  and  data.  It’s  beyond  the  scope  of  this  memo  to  show  this  original 
work,  but  I  will  later  show  supporting  evidence. 

As  a  result,  we  felt  that  if  we  could  determine  how  high  or  low  the  ratio  for  a  given  field 
image  was  in  comparison  with  the  clear  day  library  ratio,  we  could  correct  our  clear  day 
background  ratio  accordingly.  By  adjusting  the  clear  sky  background  up  or  down 
according  to  haze  amount,  we  should  be  able  to  use  the  resulting  perturbation  ratios  over 
the  image  to  more  accurately  represent  the  occurrence  of  thin  clouds. 
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4/27  1700  without  adaptive  feature  4/27  1700  with  adaptive  feature 

Fig.  10.  Cloud  Decision  Algorithm  results  for  the  same  cases  as  shown  in  Figs.  8  and  9.  The  results  with 
the  previous  algorithm  are  shown  on  the  left,  and  the  adaptive  algorithm  results  are  shown  on  the  right. 
Blue  is  where  the  algorithm  identified  no  cloud,  and  yellow  is  where  the  algorithm  identified  thin  cloud. 

To  evaluate  the  behavior  of  the  sky  ratios  as  a  function  of  haze,  we  first  evaluated  the 
ratios  at  two  specific  points  in  the  sky.  In  extracting  the  clear  sky  background,  we  always 
normalize  the  ratios  based  on  what  we  call  the  “beta  beta”  points.  These  are  two  points 
that  are  at  a  scattering  angle  p  that  is  45  degrees  from  the  sun,  and  also  at  a  zenith  angle  9 
of  45  degrees.  The  locus  of  points  with  0=45°  is  a  ring  in  the  image,  half  way  to  the 
horizon.  The  locus  of  points  with  P  =45°  is  a  ring  around  the  sun.  These  two  rings 
intersect  at  two  points.  As  part  of  the  processing  to  generate  the  clear  sky  background 
library,  we  take  the  average  of  the  Red/Blue  ratio  at  these  two  points,  and  use  it  as  our 
normalization  value  for  each  image  used  in  the  library.  (When  we  use  the  tenn 
“Red/Blue”,  we  mean  either  red/blue  or  NIR/blue,  depending  on  how  the  algorithm  was 
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set  up.)  That  is,  the  clear  sky  ratios  for  that  image  are  divided  by  the  normalization 
constant  determined  at  the  beta  points.  In  a  later  step  which  is  part  of  generating  the  clear 
sky  background  library,  all  the  normalized  ratios  that  are  within  ±  1°  of  a  solar  zenith 
angle  (SZA)  increment  of  5°  are  averaged  to  represent  that  solar  zenith  angle. 

In  processing  field  data,  when  the  clear  sky  background  for  a  given  field  image  is 
regenerated  from  the  clear  sky  library,  it  is  interpolated  between  the  nearest  two  SZA 
values  to  the  correct  SZA  value,  and  then  multiplied  by  the  selected  normalization 
constant.  The  normalized  values  that  are  stored  in  the  library  are  only  extracted  at  every 
5°  zenith  angle  and  15°  azimuth  angle,  so  when  the  clear  sky  background  for  a  given  field 
image  is  generated,  the  program  also  interpolates  between  these  look  angles. 

Fig.  1 1  shows  the  magnitude  of  the  average  ratio  at  the  two  beta  points,  for  a  number  of 
reasonably  cloud-free  (both  clear  and  hazy)  days,  and  as  a  function  of  solar  zenith  angle. 
It  rises  very  strongly  near  sunrise  and  sunset,  corresponding  with  the  whiter  (or  redder) 
skies  we  encounter  at  these  times.  In  Fig  11,  there  are  3  best  fit  curves.  In  the  previous 
version  of  the  algorithm,  we  could  choose  only  a  single  best  fit  curve  to  use  for  the  whole 
data  set.  As  a  result,  the  changes  in  the  normalization  constant  as  a  function  of  SZA  were 
already  well  handled,  but  the  day-to-day  changes  resulting  from  changing  haze  amount 
were  not  handled. 

AF5  Clear  Sky  Reference  Values  vs.  SZA:  Red/Blue  Data 
Solid  Lines  Are  Model  Curves 
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Fig.  11.  Reference  Values,  or  Normalization  Values,  for  clear  sky  ratios  during  several  cloud-free  (or 
nearly  cloud-free)  days  at  Site  5. 
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Note  that  in  general  if  a  given  image  has  a  higher  or  lower  reference  value  than  average, 
it  tends  to  stay  that  way,  to  a  first  approximation,  throughout  a  day.  That  is,  the  green 
curves  all  stay  somewhere  near  the  upper  curve,  and  the  blue  curve  stays  near  the  middle 
curve,  with  occasional  exceptions  as  demonstrated  by  the  bottom  orange  curve.  This 
implies  that  in  general  the  haze  amount  tends  to  persist  for  at  least  a  few  hours. 

8.1.4.  How  the  Adaptive  Algorithm  Works 

As  described  in  Section  4,  we  felt  that  if  we  could  find  a  way  to  predictably  determine 
how  much  higher  or  lower  the  reference  curve  should  be  on  a  given  day,  we  could  adjust 
for  the  haze.  We  wanted  this  to  be  able  to  run  in  the  field  in  real  time,  which  means  we 
couldn’t  look  at  the  whole  day  after  the  fact,  but  had  to  use  the  current  image  or  recent 
images.  We  wanted  the  correction  to  be  tested  for  each  image,  in  case  the  haze  amount 
changed  during  the  day.  The  algorithm  should  also  be  flexible  enough  to  work  well  if  a 
correction  cannot  be  detennined  for  a  specific  image. 

An  outline  of  the  adaptive  algorithm  is  as  follows.  First  we  first  create  a  line  between  the 
two  beta  points.  Then  we  compute  the  perturbation  ratio  along  this  line.  The 
perturbation  ratio  is  the  current  ratio  divided  by  the  clear  sky  ratio,  and  it  shows  how 
much  the  current  situation  is  changed  from  the  clear  sky  library  condition.  We  look  at 
the  Standard  Deviation  (STD)  of  the  perturbation  ratio  along  the  line.  If  the  STD  is  high, 
then  the  region  is  probably  not  cloud-free,  and  is  not  used  to  adjust  the  background.  But 
if  the  STD  is  low,  and  it  meets  certain  other  criteria,  then  it  is  probably  cloud-free,  and 
we  detennine  the  correction  from  the  average  perturbation  ratio  along  the  line.  This 
correction  is  then  multiplied  by  the  full  image  clear  sky  background  ratio  to  create  the 
modified  clear  sky  background  ratio  for  that  image.  And  then  the  cloud  algorithm  makes 
its  normal  checks  with  respect  to  this  adjusted  background. 

As  part  of  the  check,  we  also  verify  that  the  ratio  along  the  line  is  not  too  close  to  the 
opaque  threshold,  nor  too  close  to  the  sun.  The  more  specific  logic  of  this  part  of  the 
adaptive  algorithm  is  as  follows: 

1 .  If  the  STD  of  the  perturbation  ratio  along  the  line  is  sufficiently  low,  and  the  field 
ratios  are  not  too  high,  then  we  extract  the  correction  from  the  average  of  the  perturbation 
along  the  line. 

2.  If  the  STD  is  too  high,  we  look  at  segments  along  the  line.  If  there  are  any  good 
segments  that  meet  the  requirements,  then  we  determine  a  correction  from  the  average  of 
these  segments. 

3.  If  we  are  not  successful,  we  use  the  most  recent  correction,  if  there  is  one.  Otherwise, 
the  reference  correction  value  of  1  is  used. 

The  above  description  is  somewhat  of  a  simplification,  but  it  is  logically  correct. 
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8.1.5.  Typical  Perturbation  Values  along  the  Beta  Line 

Fig.  12  shows  a  sample  when  the  day  was  slightly  clearer  than  the  nominal  clear  day. 
The  green  curve  shows  the  magnitude  of  the  field  image  red/blue  ratio  along  the  line,  and 
the  purple  curve  shows  the  magnitude  of  the  clear  sky  library  ratio  along  the  line.  The 
algorithm  chose  a  correction  factor  of  0.84  for  this  case. 
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Current  ratio  and  Clear  sky  library  ratio  along  Beta  line 
Clear  Day,  correction  Factor  for  2/27  at  1700  was  0.84 

Fig.  12.  Sample  results  for  an  image  on  a  clear  day  with  low  haze  amount 


In  Fig.  13,  we  show  a  case  for  a  very  hazy  day.  The  field  red/blue  ratio  was  significantly 
higher  than  the  clear  sky  library  ratio  along  the  line,  and  a  correction  of  1 .42  was  chosen 
by  the  algorithm. 


Fig.  14  shows  a  case  in  which  there  were  clouds  along  some  of  the  beta  line.  Part  of  the 
beta  line  was  reasonably  clear,  however  the  STD  was  a  bit  too  high,  and  it  used  a 
correction  of  1.35  extracted  two  hours  previously.  However,  from  the  plot  we  can  see 
that  the  magnitude  of  the  correction  is  about  right,  and  we  can  see  that  the  opaque  clouds 
on  part  of  the  line  create  very  large  perturbations,  which  is  why  they  need  to  be  ignored. 
The  fact  that  the  opaque  ratio  is  so  different  from  the  clear  sky  ratio  is  one  reason  that  the 
opaque  detection  is  so  robust,  even  as  haze  amount  varies. 


The  next  question  is  how  well  the  haze  correction  factor  determined  from  the  beta  line 
applies  to  the  whole  image.  In  Figs.  15  and  16,  we  have  shown  the  field  ratio  and  the 
clear  sky  ratio,  both  for  the  beta  line,  and  for  a  line  going  through  the  sun.  Fig.  15  shows 
a  relatively  low-haze  day,  and  Fig.  16  shows  a  relatively  hazy  day. 
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Current  ratio  and  Clear  sky  library  ratio  along  Beta  line 
Hazy  day,  correction  Factor  for  4/27  at  1700  was  1.42 

Fig.  13.  Sample  results  for  a  an  image  on  a  cloud-free  day  with  high  haze  amount 
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Fig.  14.  Sample  results  for  an  image  with  clouds  along  part  of  the  beta  line 
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Measured  ratio  and  clear  sky  ratio 
For  Beta  line  (left)  and  row  through  the  sun  (right) 
For  clear  day,  2/27/08 


Fig.  15.  Red/blue  ratio  for  the  current  image  (green  line)  and  its  nominal  clear  sky  background  (purple 
line).  Plot  on  left  is  for  beta  line,  and  plot  on  right  is  for  line  through  the  sun.  This  is  for  a  relatively  low 
haze  case. 
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Fig.  16.  Red/blue  ratio  for  the  current  image  (green  line)  and  its  nominal  clear  sky  background  (purple 
line).  Plot  on  left  is  for  beta  line,  and  plot  on  right  is  for  line  through  the  sun.  This  is  for  a  relatively  high 
haze  case. 
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As  can  be  seen  in  Figs.  15  and  16,  the  separation  between  the  green  and  purple  lines  for 
the  beta  line  (plot  on  the  left)  appears  to  be  reasonably  representative  of  the  separation 
that  occurs  on  a  line  through  the  sun  and  the  horizon.  That  is,  the  perturbation  ratio  along 
the  beta  line  appears  to  represent  the  perturbation  for  other  parts  of  the  image  quite  well. 
This  indicates  that  the  haze  correction  determined  using  the  beta  line  should  apply 
reasonably  well  to  the  full  image.  We  found  that  this  was  the  case;  that  is,  the  correction 
works  quite  well  over  the  whole  image  in  most  cases. 

8.1.6.  Sample  Results 

Examples  for  clear  skies,  both  with  and  without  heightened  haze,  have  already  been 
shown  in  Figs.  8-10.  Note  in  the  bottom  right  image  of  Fig.  10  that  there  is  a  grey  circle 
around  the  sun.  This  is  a  region  characterized  as  “indeterminate”.  In  this  case,  the  clear 
sky  background,  as  adjusted  for  the  high  haze,  has  a  high  enough  ratio  that  it  cannot 
readily  be  separated  from  opaque  cloud.  In  both  of  these  clear  cases,  the  algorithm  did 
very  well.  The  horizon  was  very  slightly  worse  in  both  cases,  but  the  sponsors  have  told 
us  that  they  no  longer  care  about  the  horizon,  so  we  felt  that  it  was  not  cost-effective  to 
try  to  improve  the  horizon. 

The  results  both  with  and  without  the  adaptive  feature  of  the  algorithm  for  a  case  with 
broken  clouds  are  shown  in  Fig.  17.  Other  sample  results  will  be  shown  in  the  sections 
discussing  the  processing  of  the  field  data. 

The  Site  5  Jan  -  May  data  set  was  used  to  develop  the  adaptive  algorithm.  To  further  test 
the  results  of  the  adaptive  algorithm,  we  ran  an  additional  run  with  the  same  input 
parameters,  but  with  the  adaptive  feature  of  the  algorithm  turned  off.  We  evaluated  the 
hourly  data,  and  found  that  out  of  1401  cases  we  evaluated,  approximately  24%  of  the 
cases  were  significantly  better  with  the  adaptive  algorithm  turned  on.  Approximately 
74%  were  good  with  either  algorithm,  and  approximately  2%  of  the  cases  were 
significantly  worse  with  the  adaptive  algorithm  turned  on.  Most  of  the  cases  in  which  the 
algorithm  did  better  were  in  April  and  May,  and  were  the  heavy  haze  cases.  Examples 
are  shown  in  Fig.  10.  Most  of  the  cases  in  which  the  algorithm  did  worse  were  cases  in 
which  there  was  cirrus,  and  the  algorithm  did  not  pick  up  as  much  of  the  cirrus,  although 
it  generally  picked  up  some. 

In  summary,  the  new  adaptive  algorithm  evaluates  the  images  on  a  case  by  case  basis,  to 
adjust  for  the  impact  of  haze  on  the  image.  If  it  cannot  make  a  current  determination,  it 
uses  the  most  recent  determination,  as  corrected  for  changing  solar  zenith  angle.  We 
found  that  the  new  algorithm  is  not  perfect,  but  is  much  better  than  the  previous  version, 
in  that  it  avoids  calling  heavy  haze  conditions  thin  cloud. 
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Result  without  adaptive  feature  of  algorithm  Result  with  adaptive  algorithm 

Fig.  17.  Sample  result  for  broken  clouds,  9  May  08  1700 


8.2.  Site  5  Daytime  Processing  and  Real  Time  Algorithm 

Under  this  contract,  the  archival  Daytime  Site  5  data  for  January  16  -  May  20  2008  were 
processed  and  analyzed  and  delivered.  The  instrument  was  redeployed  on  January  16 
2008,  following  repair  from  a  lightning  strike  in  2007.  The  instrument  was  down  from 
May  2 1  through  July  1  due  to  an  occultor  failure.  The  post-repair  data  were  evaluated, 
and  the  inputs  extracted  to  run  the  algorithm  in  real  time.  The  real  time  algorithm  was 
installed  on  the  system  on  October  14,  2008.  The  details  of  the  analysis  are  documented 
in  Memo  AV09-042t.  The  inputs  for  the  real  time  algorithm  are  documented  in  Memo 
AV09-043,  which  also  documents  the  processing  of  the  post-repair  data,  which  was  done 
under  the  follow-on  contract.  It  should  be  noted  that  the  documentation  of  these  data  was 
jointly  funded  with  the  follow-on  contract. 
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Sample  results  for  two  clear  days  discussed  in  Section  8.1  are  repeated  in  Figs.  18  and  19. 
Additional  sample  results  are  shown  in  Figs.  20  -  22.  We  are  very  pleased  with  these 
results.  Fig.  22  shows  an  excellent  case  with  cirrus  fonning  from  contrails,  and 
demonstrates  the  ability  of  the  WSI  to  detect  contrails. 


Fig.  18.  Raw  image  and  Adaptive  Algorithm  results  for  clear  case,  2/27/08  at  1700  GMT 


Fig.  19.  Raw  image  and  Adaptive  Algorithm  results  for  clear  case  with  haze,  4/27/08  at  1700  GMT 

The  rest  of  the  Site  5  Daytime  data  was  processed  under  the  follow-on  contract,  and  will 
be  discussed  in  the  next  report.  The  format  of  the  archival  delivery  is  documented  in 
Technical  Memo  AV08-037t,  repeated  in  Appendix  1.  The  use  of  the  archival  data, 
including  the  equations  for  the  geometric  calibration,  is  documented  in  Technical  Memo 
AV08-038t,  repeated  in  Appendix  2.  The  use  of  the  real  time  data  is  documented  in 
Technical  Memo  AV09-024t. 
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Fig.  20.  Raw  image  and  Adaptive  Algorithm  results  for  scattered  cloud  case,  5/4/08  at  1700  GMT 


Fig.  21.  Raw  image  and  Adaptive  Algorithm  results  for  broken  cloud,  1/24/08  at  1700  GMT 


Fig.  22.  Raw  image  and  Adaptive  Algorithm  results  for  cirrus  forming  from  contrails,  1/20/08  at  2000 
GMT 
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8.3.  Site  3  Daytime  Processing  and  Real  Time  Algorithm 

During  the  time  of  this  contract,  the  archival  Daytime  Site  3  data  for  April  9  -  August  4 
were  processed  and  analyzed  and  delivered.  The  instrument  was  deployed  during  8-10 
April  2008.  The  real  time  algorithm  was  installed  on  the  system  on  October  14,  2008 
(the  same  day  as  for  Site  5).  It  should  be  noted  that  the  analysis  and  documentation  of 
these  data  was  jointly  funded  with  the  follow-on  contract. 

The  details  of  the  analysis  are  documented  in  Memo  AV09-008t.  These  data  were 
somewhat  spotty  due  to  occultor  and  cooler  problems,  however  both  problems  were 
repaired.  Data  marked  “Uncertain”  in  the  data  header  should  not  be  used.  If  this 
precaution  is  taken,  then  the  processed  data  may  be  used  with  good  confidence,  as  it 
eliminates  the  cases  impacted  by  the  heat  and  occultor. 

Sample  results  for  a  clear  day  and  a  scattered  cloud  case  are  shown  in  Figs.  23  and  24. 
Fig.  25  shows  a  series  of  hourly  images  taken  over  6  hours,  that  illustrate  that  the 
algorithm  generally  works  well  under  all  conditions.  One  can  also  note  in  this  figure  that 
when  there  were  very  thin  cirrus,  the  algorithm  may  not  identify  them.  In  general,  when 
this  happens  the  algorithm  will  identify  some  of  the  cirrus,  but  not  all  of  them.  This 
happens  because  the  thin  cloud  optical  density  threshold  is  not  0;  that  is,  some  very  thin 
clouds  will  be  missed.  Our  sponsors  felt  that  it  is  reasonable  to  have  a  threshold  of 
around  1  dB  (transmittance  .79).  We  have  estimated  the  optical  density  thresholds  will  at 
night,  as  will  be  discussed  further  in  Section  9.  Under  the  follow-on  contract,  we  began 
developing  methods  to  quantify  the  thin  cloud  optical  density  associated  with  the  thin 
cloud  threshold  for  the  daytime  algorithm,  but  then  we  were  asked  not  to  make  this  a 
priority. 
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Fig.  24.  Mixed  cloud  results,  22  May  08  2100. 


Fig.  25.  Flourly  time  series,  23  May  08  1400  -  1900. 


The  rest  of  the  Site  3  Daytime  data  was  processed  under  the  follow-on  contract,  and  will 
be  discussed  in  the  next  report.  The  format  of  the  archival  delivery  is  documented  in 
Technical  Memo  AV08-037t,  repeated  in  Appendix  1.  The  use  of  the  archival  data, 
including  the  equations  for  the  geometric  calibration,  is  documented  in  Technical  Memo 
AV08-038t,  repeated  in  Appendix  2.  The  use  of  the  real  time  data  is  documented  in 
Technical  Memo  AV09-024t. 
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8.4.  Summary  of  the  Day  Algorithm  and  Accuracy  Test  Results 

The  adaptive  feature  of  the  daytime  cloud  algorithm  was  developed  under  this  contract, 
and  is  working  very  well.  Its  function  is  to  adjust  the  algorithm  for  changes  in  the  aerosol 
or  haze,  so  that  high  aerosol  (enhanced  number  of  small  droplets)  will  not  be  identified  as 
thin  cloud.  Typically,  real  thin  cloud  consists  of  larger  droplets,  and  is  often  much  more 
variable  spatially  than  haze.  We  prefer  to  identify  thin  cloud  as  cases  with  enhanced 
droplet  size,  because  they  will  impact  the  short  wave  IR  as  will  as  the  visible. 

Under  this  contract,  we  were  able  to  get  the  real  time  algorithm  in  the  field  for  Sites  3  and 
5,  and  process  part  of  the  archival  data.  Site  2  already  had  a  working  version  of  the 
daytime  algorithm  running  in  real  time  (although  it  did  not  include  the  adaptive 
algorithm).  The  update  to  Site  2  was  not  required  under  this  contract,  and  it  was 
completed  under  the  follow-on  contract.  In  a  test  case,  we  found  that  the  cloud  algorithm 
looked  good  either  with  or  without  the  adaptive  feature  74%  of  the  time.  Results  were 
improved  with  the  adaptive  algorithm  24%  of  the  time,  and  not  as  good  2%  of  the  time. 

At  this  point  we  believe  the  biggest  strength  of  the  day  cloud  algorithm  is  that  it’s  nearly 
always  correct  for  clear  and  opaque  cases.  It  also  nearly  always  identifies  most  of  the 
cirrus  clouds.  It  does  have  a  non-zero  threshold  for  thin  cloud,  in  that  it  does  not  identify 
the  thinnest  of  the  cirrus.  Probably  the  biggest  weakness  of  the  algorithm  is  that  we  have 
not  yet  quantified  the  optical  depth  associated  with  this  threshold. 

In  tests  done  under  the  follow-on  contract,  we  wrote  a  program  for  doing  “blind”  tests,  to 
assess  the  accuracy  of  the  algorithm.  In  these  tests,  we  identified  thin  cirrus  as 
“uncertain”,  because  we  were  not  sure  whether  they  should  be  above  or  below  the 
threshold.  We  also  identified  certain  other  cases  as  uncertain,  including  small  holes  in 
opaque  clouds,  where  there  can  be  cloud  debris,  or  small  cloud  fragments,  that  are  not 
always  obvious  in  the  raw  imagery.  Of  those  cases  that  were  not  called  uncertain,  we 
achieved  a  99.3%  accuracy  rate.  The  blind  test  will  be  documented  in  the  next  report. 

As  a  result  of  this  analysis,  we  felt  that  the  day  adaptive  algorithm  was  very  accurate  and 
under  the  follow-on  contract,  we  worked  primarily  on  extracting  and  processing 
additional  archival  data. 

9.  Night  Algorithm  Developments  and  Analysis 

During  the  previous  contract,  as  documented  in  Technical  Note  273,  we  developed  a  new 
high  resolution  night  algorithm.  Our  previous  algorithm  provided  a  result  in  each  of  357 
cells  over  the  sky,  whereas  the  new  algorithm  provides  an  answer  at  each  pixel. 
(Although  the  result  is  at  full  pixel  resolution,  a  5  x  5  smoothing  of  the  image  is  used 
during  part  of  the  processing,  so  we  refer  to  it  as  a  “high  resolution”  algorithm.)  It  is 
based  on  measurement  of  the  earth-to-space  beam  transmittance  determined  from  the 
stars  in  the  image,  and  then  the  absolute  radiance  of  the  image  is  used  to  extend  it  to  high 
resolution. 
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At  the  end  of  the  previous  contract,  the  high  resolution  night  algorithm  was  working  very 
well  for  starlight  conditions,  however  we  had  not  yet  developed  good  logic  for  moonlight 
conditions.  Under  the  grant  which  is  the  subject  of  this  report,  we  completed  the 
development  of  the  moonlight  algorithm,  and  fielded  the  new  algorithm  at  Sites  3  and  5, 
as  required  by  the  contract.  During  the  interval  of  this  contract,  we  also  processed  and 
delivered  the  archival  data  for  these  sites,  however  the  archival  processing  was  mostly 
funded  under  the  follow-on  contract. 

9.1.  The  High  Resolution  Algorithm  Concept 

We  have  already  discussed  the  development  of  the  high  resolution  starlight  algorithm  in 
some  detail  in  Technical  Notes  272  and  273.  Therefore  we  only  provide  a  brief  overview 
in  this  section  of  how  the  starlight  algorithm  works. 

9.1.1  Transmittance 

The  concept  behind  the  high  resolution  algorithm  was  originally  tested  and  documented 
in  Memo  AV01-069t.  This  algorithm  requires  determination  of  the  absolute  value  of 
earth  to  space  beam  transmittance  for  the  stars.  As  discussed  in  Technical  Note  271,  we 
developed  the  ability  to  determine  the  absolute  value  of  earth  to  space  beam 
transmittance  for  stars,  before  working  further  on  the  high  resolution  algorithm.  On  clear 
nights,  we  were  able  to  obtain  a  correlation  of  0.937  between  computed  inherent  star 
irradiance  and  measured  apparent  star  irradiance  on  cloud-free  nights,  where  “inherent” 
means  above  the  atmosphere,  and  “apparent”  means  as  measured  on  the  ground.  This 
number  was  for  a  data  comparison  of  43,000  stars,  down  to  magnitude  6,  as  documented 
in  Memo  AV07-037t. 

In  our  next  contract,  documented  in  Technical  Note  272,  we  improved  the  determination 
of  earth  to  space  beam  transmittance,  and  did  some  studies  of  the  accuracy  of  the  results. 
We  found  that  the  results  made  sense,  in  that  the  clear  sky  results  were  reasonable  for 
aerosol  transmittance,  and  the  decreases  in  transmittance  were  associated  with  visually 
detected  increases  in  cloud  thickness  in  the  imagery.  We  were  able  to  detennine  the 
transmittance  over  a  wide  range  down  to  about  a  transmittance  of  .01,  and  found  that  the 
accuracy  was  about  5%  at  the  high  end,  although  there  will  be  rare  “glitch”  values  in  the 
clouds  that  are  not  correct.  We  felt  that  these  results  were  sufficiently  accurate  for 
identifying  the  presence  of  opaque  and  thin  clouds. 

9.1.2.  Starlight  Methods 

As  discussed  in  Technical  Note  272,  we  also  further  tested  the  concepts  for  a  high 
resolution  algorithm.  The  concept  is  basically: 

a)  Calibrate  the  night  images  for  absolute  radiance. 

b)  Detennine  the  typical  shape  of  the  clear  sky  radiance  distribution  under  starlight.  Call 
this  the  “clear  shell”. 
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c)  Determine  the  typical  shape  of  the  opaque  cloud  distribution  under  starlight.  Call  this 
the  “cloud  shell”. 

d)  Using  the  earth-to-space  beam  transmittance,  identify  the  likely  cloud  condition  in  the 
direction  of  the  stars  as  either  no  cloud,  thin  cloud,  or  opaque  cloud. 

e)  Use  the  radiance  at  those  points  identified  as  no  cloud  to  adjust  the  clear  shell  for 
current  conditions. 

f)  Use  the  radiance  at  those  points  identified  as  opaque  cloud  to  adjust  the  opaque  shell 
for  current  conditions. 

g)  At  pixels  not  associated  with  stars,  compare  the  measured  radiance  with  the  adjusted 
clear  shell  and  cloud  shell  to  assess  whether  that  pixel  has  no  cloud,  thin  cloud,  or  opaque 
cloud. 

This  concept  for  the  high  resolution  algorithm  was  developed  for  the  starlight  case,  and  is 
documented  in  Technical  Note  273.  This  initial  work  was  done  using  the  data  from  Site 
2.  Both  the  clear  and  cloud  shells  were  determined  from  the  in-situ  data.  It  was 
necessary  to  sort  the  clear  data  as  a  function  of  hour  angle.  This  site  is  near  a  city,  and 
fairly  brightly  lit.  We  first  found  that  there  was  a  good  separation  between  the  clear  and 
opaque  shells,  which  means  that  the  algorithm  concept  should  work  well.  Fig.  26  was 
extracted  from  TN  273,  and  illustrates  the  extracted  clear  and  cloud  shells  for  both  the 
SOR  site  and  Site  2. 

For  a  given  site,  the  background  imagery  used  to  create  the  shells  for  the  night  algorithm 
is  obtained  by  compiling  many  images  that  visually  appear  to  be  cloud-free,  and 
determining  the  median  value  of  calibrated  radiance  at  each  pixel.  The  data  were  sorted 
into  several  hour  angle  ranges  and  averaged,  so  there  are  typically  13  background  images 
for  a  site  to  cover  the  range  of  possible  hour  angles. 


SOR  Hour  Angle  Bkgrnd  And  Smoothed  Opaque  Cloud  AF 2  Hour  Angle  Bkgrnd  And  Smoothed  Opoque  Cloud 

Cross-Section:  Middle  Row  Cross-Section:  Middle  Column 


Fig.  26.  Typical  no-moon  clear  sky  radiance  levels  for  clear  sky  as  a  function  of  hour  angle  (solid  color 
curves),  and  typical  no-moon  cloud  radiance  levels  (dash  line).  Plot  on  left  is  for  SOR  site,  and  plot  on 
right  is  for  Site  2.  Left  plot  y-axis  scale  is  0  to  1.5  10'4.  Right  plot  y  axis  scale  is  0  to  9  10"4. 

A  similar  technique  is  used  to  determine  the  opaque  shell,  although  we  did  not  find  it 
necessary  to  sort  the  opaque  shell  as  a  function  of  hour  angle.  A  discussion  of  the 
background  image  determination  may  be  found  in  Tech.  Memo  AV05-012t. 
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A  method  was  also  developed  to  adjust  the  clear  and  cloud  shells,  as  outlined  in  steps  e 
and  f  above.  For  each  star  within  60°  of  image  center  identified  as  not  being  obscured  by 
cloud,  a  correction  factor  was  determined  from  the  ratio  of  the  background  radiance  to 
the  shell  radiance  at  that  position.  The  average  correction  factor  for  that  image  is  applied 
to  the  shell  for  that  image.  If  no  usable  stars  are  found  using  the  above  criteria,  a 
correction  from  an  earlier  image  is  applied.  In  addition,  the  clear  shell  is  moved  15% 
higher.  In  this  way,  the  radiance  at  a  given  pixel  must  exceed  the  adaptively  adjusted 
clear  shell  by  15%  in  order  to  be  identified  as  thin  cloud.  The  15%  is  an  input  parameter 
that  can  be  modified.  To  more  accurately  identify  thin  cloud  (at  the  possible  expense  of 
misidentifying  some  cloud-free  regions  as  thin  cloud),  the  factor  may  be  adjusted  down. 

The  approach  is  similar  for  the  opaque  shell.  Also,  the  cloud  shell  is  further  adjusted 
using  a  multiplier  of  .90,  so  that  the  cloud  shell  should  be  below  the  actual  measured 
cloud  radiances. 

Some  of  the  details  are  provided  in  Technical  Note  273,  and  more  details  are  provided  in 
Memo  AV06-029t.  Examples  are  given  in  Technical  Note  273.  The  overall  results  were 
about  99%  accurate  at  the  zenith,  and  94%  accurate  over  the  whole  image.  I  note  that 
there  is  an  error  in  Figure  35  of  Technical  Note  273.  Since  this  is  an  important  figure, 
showing  the  accuracy  of  the  high  resolution  algorithm  for  no  moon,  I  will  show  the 
corrected  figure  in  Fig.  27. 


Fig.  27.  Fraction  of  correct  answers,  in  percent,  for  each  Line  of  Sight  for  Site  2  Test  Data  using  Program 
SORCloudAssess,  for  no  moon  cases.  This  is  the  corrected  version  of  Fig.  35  from  Technical  Note  273. 

As  this  project  developed,  our  sponsors  decided  they  were  less  interested  in  accuracy  near 
the  horizons.  For  those  zenith  angles  of  60  degrees  and  higher,  the  results  shown  above 
average  95.8%.  Sample  results  are  shown  in  Technical  Note  273.  We  were  very  pleased 
with  these  results. 
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9.2.  Development  of  the  Moonlight  High  Resolution  Night  Algorithm 


As  discussed  in  Memo  AV09-056t,  it  is  considerably  more  difficult  to  develop  the  clear 
and  cloud  shells  for  moonlight,  for  a  number  of  reasons.  One  is  that  there  are  fewer 
moonlight  cases  with  full  moon,  so  it  is  difficult  to  find  clear  and  cloudy  cases.  Another 
is  that  it  is  necessary  to  contend  with  changes  in  the  phase  of  the  moon.  Under  clear  sky 
with  a  full  moon  the  radiance  distribution  is  primarily  due  to  a  sum  of  the  light  scattered 
from  the  moonlight,  and  the  light  from  man-made  or  anthropogenic  sources. 

We  used  the  approach  of  modeling  the  shape  of  the  scattered  portion  of  the  moonlight 
imagery.  To  do  this,  we  first  modeled  the  moonlight  sky  using  the  equation: 

N(0, </),  0M ,  (J) M )  —  N*  (0,  <f>)  +  RN(a)  *  NM  (9, <f>,  0U , (j)M ,  amax )  (1) 


where 

N(9,<f>,9M,<f>M)  is  the  radiance  distribution  at  night  as  a  function  of  look  angle  9,(j)  and 
the  Source  (Moon)  position  9U ,  (j)M 
N g  {9,  (ft)  is  the  radiance  distribution  of  the  moonless  background  sky 
RN(a )  is  the  relative  brightness  of  the  moon  as  a  function  of  phase  angle  a 
NM(9,(f),9M  ,(j>M  ,amax)  is  the  radiance  distribution  of  the  full-moon-lit  sky  as  a  function 
of  look  angle  9 ,  (f)  and  the  Source  (Moon)  position  9U ,  (j)M  and  phase  angle. 

Using  this  method,  the  radiance  distribution  of  the  no-moon  sky  NB  {9,  (j))  would  be  the 
same  as  previously  extracted.  The  relative  brightness  of  the  moon  RN(a )  is  computed 
as  a  function  of  phase  angle  in  our  program  Occlnfo,  and  takes  into  account  both  phase 
and  earth-to-moon  distance  to  determine  relative  brightness. 

Programs  were  written  to  enable  us  to  extract  the Nu(9,<j), a) radiance  distribution 

of  the  moonlight  sky  from  night  calibrated  radiance  images.  The  programs  are  similar  to 
those  used  to  extract  the  red/blue  ratio  distribution  for  the  day  algorithm. 

When  extracting  the  distribution  for  a  given  moonlight  case,  the  image  is  first  corrected 
for  the  background  radiance  that  occurs  under  starlight.  Also,  the  radiance  distribution  is 
corrected  for  the  relative  moon  brightness.  Once  the  clear  sky  library  has  been 
determined  for  a  nominal  relative  moon  brightness  of  1,  then  the  radiance  for  a  specific 
case  can  be  determined  using  Eqn.  1 . 

As  with  the  no-moon  portion  of  the  algorithm,  stars  are  used  to  identify  portions  of  the 
sky  with  no  cloud  and  opaque  cloud,  and  the  radiance  in  the  direction  of  these  stars  is 
used  to  compute  an  average  correction  factor.  When  corrected,  the  clear  shell  should 
match  the  current  clear  sky  portions  of  an  image  reasonably  well,  before  the  additional 
15%  correction  is  applies. 
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In  terms  of  the  cloud  shell,  we  found  that  the  cloud  shell  used  during  no  moon  could  also 
be  used  for  moonlight  cases.  As  with  the  no-moon  algorithm,  the  algorithm  adjusts  the 
cloud  shell  using  the  radiance  at  the  stars  that  are  identified  as  being  blocked  by  opaque 
clouds.  In  this  way,  the  magnitude  of  the  cloud  shell  is  adjusted  from  image  to  image, 
but  the  shape  of  the  cloud  shell  for  moonlight  is  the  same  as  the  shape  of  the  cloud  shell 
for  starlight.  Also,  the  cloud  shell  is  further  adjusted  using  a  multiplier  of  0.90. 

To  determine  the  algorithm  result  for  a  given  pixel,  if  the  radiance  is  higher  than  the 
adjusted  cloud  shell,  it  is  called  opaque  cloud.  If  it  is  lower  than  the  adjusted  clear  shell, 
it  is  called  no  cloud.  If  it  is  between,  it  is  called  thin  cloud.  If  a  region  occurs  in  which 
the  adjusted  clear  shell  is  higher  than  the  adjusted  cloud  shell,  this  region  is  identified  as 
indeterminate.  The  software  also  includes  a  feature  that  allows  the  adjusted  shell 
correction  determined  from  one  processed  image  to  be  applied  to  other  images  later  in  the 
same  day  when  no  usable  stars  are  found  in  those  images. 

9.3.  Sample  Results  for  Site  5 

A  typical  clear  case  with  moon  and  the  algorithm  result  is  shown  in  Fig.  28.  In  this 
example,  the  moon  is  at  .61  relative  brightness. 


Fig.  28.  Raw  Image  and  Cloud  Algorithm  Results  for  Site  5,  2  June  2007,  0956  GMT 


The  algorithm  results  for  the  image  shown  in  Fig.  28  is  excellent.  In  this  image,  blue 
indicates  that  the  algorithm  estimates  there  to  be  no  cloud,  green  indicates  thin  cloud,  and 
grey  indicates  indeterminate,  as  defined  in  Section  3.  Opaque  cloud,  when  it  occurs,  is 
indicated  with  white  in  the  algorithm  result. 

The  modeled  values  for  the  clear  sky  radiance  and  moon  radiance  for  this  case  are  shown 
in  Fig.  29,  for  a  vertical  column  at  x  pixel  255  (in  the  center  of  the  sensor  image).  On  the 
left  hand  side,  the  red  is  the  starlight  clear  sky  background  for  the  hour  angle  of  the 
image,  and  the  green  is  the  moonlight  model  clear  sky  without  the  starlight  radiance,  for 
the  moon  position  and  relative  brightness  of  this  image.  The  blue  is  the  combined  model 
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radiance  to  yield  the  modeled  clear  sky  radiance  for  this  image.  When  the  starlight  and 
moonlight  radiances  are  combined  to  show  the  modeled  total  radiance  in  blue,  we  see  that 
it  compares  well  with  the  measured  radiance,  shown  in  black.  Fig.  29  includes  the 
adaptive  correction,  which  is  a  factor  of  0.903  for  this  sample.  We  want  the  model  to 
exceed  the  measured  by  15%,  so  that  we  in  effect  have  a  15%  threshold.  As  a  result,  an 
additional  threshold  factor  was  1.15  was  applied,  meaning  that  the  shell  image  was 
adjusted  upward  by  1.038%  for  use  with  this  particular  image.  The  curves  with  this 
correction  are  shown  on  the  right  side  of  Fig.  29. 


Row  Row 

Fig.  29.  Model  and  Measured  Radiances,  with  Adaptive  Corr  of  1.09  (left)  and  additional  15%  threshold 
correction  (right),  for  Site  5,  2  June  2007,  0956  GMT  These  plots  are  for  column  255,  i.e.  a  vertical  line 
through  the  image.  We  show  the  modeled  no-moon  radiance  in  orange,  the  modeled  moonlight  radiance  in 
green,  the  combined  model  radiance  with  the  adaptive  correction  of  0.903  in  blue,  and  the  measured  image 
radiance  in  black. 


One  of  the  reasons  for  the  1.15  factor  for  the  clear  shell  and  a  .90  factor  for  the  cloud 
shell  is  that  we  use  one  adjustment  factor  for  a  whole  image.  The  shape  of  the  shell  will 
never  be  precisely  the  same  as  the  shape  of  the  radiances  for  a  given  image.  This  simply 
means  that  the  threshold  for  thin  cloud  detection  is  not  0  db,  but  is  a  finite  but  very  low 
number.  In  later  research  that  will  be  discussed  in  the  report  for  the  follow-on  contract, 
we  found  that  the  threshold  for  optical  fade  for  the  thin  clouds  appears  to  be  about  a  loss 
of  0.7  to  1.0  dB  (transmittance  .85  to  .79).  The  threshold  for  optical  fade  for  the  opaque 
clouds  appears  to  be  a  loss  of  about  8  dB  (transmittance  .16).  We  feel  these  thresholds 
are  appropriate,  in  that  the  clouds  that  are  missed  are  quite  thin,  and  these  thresholds  help 
minimize  false  alarms  in  clear  areas. 


To  demonstrate  the  concept  for  situations  with  clouds,  we  have  chosen  a  series  with 
variable  conditions  that  happened  a  few  hours  prior  to  the  sample  shown  in  Fig.  29.  In 
Fig.  30,  we  show  a  case  about  4  %  hours  earlier,  in  which  there  were  some  extremely  thin 
clouds,  barely  distinguishable  in  the  raw  imagery.  From  the  left  plot  in  Fig.  31,  we  see 
that  the  model  and  measured  clear  sky  match  very  well  in  the  center  of  the  curve  (i.e. 
near  the  zenith),  where  the  sky  is  clear,  and  the  thin  clouds  can  be  seen  as  slight 
enhancements  on  the  parts  of  the  curve  away  from  the  center.  The  plot  with  the 
additional  15%  threshold  is  shown  in  the  right  plot  in  Fig.  31.  Much  of  this  very  thin 
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cloud  is  not  above  threshold.  The  algorithm  successfully  identifies  the  slightly  thicker 
clouds  that  are  above  this  threshold.  The  opaque  shell  is  not  shown  in  Fig.  31,  but  the 
pixels  are  well  below  the  opaque  shell.  (Examples  of  the  opaque  shell  will  be  shown 
later  in  this  section.) 


Fig.  30.  Raw  Image  and  Cloud  Algorithm  Results  for  Site  5,  2  June  2007,  0530  GMT 


Moon  Zenith  =  73.40  Moon  Azim  =  141.50  Moon  Brightness  =  0.63  Moon  Zenith  =  73.40  Moon  Azim  =  141.50  Moon  Brightness  =  0.63 

Radiances  from  Column  255  Radiances  from  Column  255 

Site:  AF5  Date:  02  Jun  2007  05:30  UTC  Site:  AF5  Date:  02  Jun  2007  05:30  UTC 
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Fig.  31.  Model  and  Measured  Radiances,  with  Adaptive  Corr  of  1.09  (left)  and  additional  15%  threshold 
correction  (right),  for  Site  5,  2  June  2007,  0530  GMT 


In  Figs.  32  and  33,  from  imagery  taken  about  an  hour  later,  we  see  that  the  clouds  are 
somewhat  thicker,  and  a  halo  has  become  visible.  As  seen  in  Fig.  33,  the  clouds  in  the 
lower  part  of  the  image  (left  side  of  x  axis)  are  slightly  higher  than  the  nominal  clear  sky, 
but  slightly  lower  than  the  shell  corrected  for  the  15%  threshold.  The  upper  part  of  the 
image  clearly  has  thicker  clouds,  and  these  are  identified  as  thin  cloud. 
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Fig.  32.  Raw  Image  and  Cloud  Algorithm  Results  for  Site  5,  2  June  2007,  0626  GMT 


Moon  Zenith  =  67.20  Moon  Azim  =  152.30  Moon  Brightness  =  0.62  Moon  Zenith  =  67.20  Moon  Azim  =  152.30  Moon  Brightness  =  0.62 

Radiances  from  Column  255  Radiances  from  Column  255 

Site:  AF5  Date:  02  Jun  2007  06:26  UTC  Site:  AF5  Date:  02  Jun  2007  06:26  UTC 
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Fig.  33.  Model  and  Measured  Radiances,  with  Adaptive  Corr  1.23  (left)  and  additional  15%  threshold 
correction  (right),  for  Site  5,  2  June  2007,  0626  GMT 


Figs.  34  and  35  show  the  measured  transmittances  for  the  stars  for  these  two  cases  at 
0530  and  0626,  with  the  key  shown  in  Fig.  36.  We  see  that  indeed  the  transmittances 
were  lower  at  0626,  and  it  is  thus  reasonable  that  the  algorithm  result  at  0626  shows  more 
cloud. 

Figs.  37  through  40  show  the  results  as  the  clouds  became  thicker,  and  then  nearly 
disappeared  except  for  a  very  thin  cloud  near  the  bottom  of  the  occultor.  In  Figs.  37  and 
38  we  see  that  part  of  the  cloud  field  has  become  fairly  thick,  and  this  is  shown  in  a  fairly 
high  spike  in  Fig.  38.  Although  the  cloud  shell  is  not  shown,  this  spike  is  lower  than  the 
cloud  shell.  Also  note  in  these  figures  that  when  a  reasonably  thick  cloud  appears,  as  in 
Fig.  38,  the  signal  is  much  higher  than  the  clear  shell,  so  the  algorithm  should  work  very 
consistently  in  identifying  the  thicker  clouds  as  being  cloud.  The  very  thin  cloud  seen  in 


42 


Fig.  39  is  also  seen  in  Fig.  40,  where  it  appears  to  be  about  20%  higher  than  the  adjusted 
clear  shell. 


Fig.  34.  Transmittance  map  for  2  June  2008  0530  Fig.  35.  Transmittance  map  for  2  June  2008  0626 
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Fig.  36.  Key  for  Transmittance  maps 


Results  for  a  moonlight  case  with  some  opaque  clouds  are  shown  in  Figs.  41  and  42.  The 
algorithm  worked  very  well.  The  plot  in  Fig.  42  shows  the  cloud  shell  as  well  as  the 
other  shells.  (We  could  have  shown  the  cloud  shells  in  the  earlier  plots,  but  didn’t  want 
to  clutter  the  plots.)  The  left  half  of  Fig.  42  has  been  adjusted  using  the  radiances  near 
the  stars  that  were  identified  as  having  opaque  cloud,  so  the  curve  roughly  matches  the 
observed  radiance.  The  .90  factor  has  been  applied  to  the  cloud  shell  in  the  right  side  of 
Fig.  42,  which  allows  the  opaque  clouds  to  be  identified  as  opaque. 
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Fig.  37.  Raw  Image  and  Cloud  Algorithm  Results  for  Site  5,  2  June  2007,  0656  GMT 


Moon  Zenith  =  64.60  Moon  Azim  =  158.70  Moon  Brightness  =  0.62  Moon  Zenith  =  64.60  Moon  Azim  =  158.70  Moon  Brightness  =  0.62 

Radiances  from  Column  255  Radionces  from  Column  255 

Site:  AF5  Date:  02  Jun  2007  06:56  UTC  Site:  AF5  Date:  02  Jun  2007  06:56  UTC 
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Fig.  38.  Model  and  Measured  Radiances,  with  Adaptive  Corr  1.13  (left)  and  additional  15%  threshold 
correction  (right),  for  Site  5,  2  June  2007,  0656  GMT 


Although  the  no-moon  algorithm  has  not  changed  appreciably,  we  would  like  to  show 
one  examples  for  no  moon  also.  Figs.  43  and  44  show  a  case  with  scattered  clouds.  We 
feel  these  results  are  excellent,  like  the  moonlight  cases. 

As  noted  earlier,  we  are  using  the  same  cloud  shell  for  both  moonlight  and  starlight 
conditions,  because  the  shape  of  the  cloud  shell  is  relatively  unchanging.  We  also  note 
that  the  adaptive  correction  for  the  cloud  shell  is  less  than  one  in  the  starlight  example 
below,  and  more  than  one  in  the  moonlight  examples  above.  This  makes  sense,  because 
the  magnitude  of  the  cloud  shell  should  depend  to  some  extent  on  the  moonlight,  and  it 
shows  the  effectiveness  of  the  adaptive  feature  of  the  algorithm. 
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Fig.  39.  Raw  Image  and  Cloud  Algorithm  Results  for  Site  5,  2  June  2007,  0750  GMT 


Moon  Zenith  =  61.70  Moon  Azim  =  171.30  Moon  Brightness  =  0.62  Moon  Zenith  =  61.70  Moon  Azim  =  171.30  Moon  Brightness  =  0.62 

Radiances  from  Column  240  Radiances  from  Column  240 

Site:  AF5  Date:  02  Jun  2007  07:50  UTC  Site:  AF5  Date:  02  Jun  2007  07:50  UTC 


Fig.  40.  Model  and  Measured  Radiances,  with  Adaptive  Corr  0.94  (left)  and  additional  15%  threshold 
correction  (right),  for  Site  5,  2  June  2007,  0750  GMT 


Fig.  41.  Raw  Image  and  Cloud  Algorithm  Results  for  Site  5,  1  Feb  2007,  1000  GMT 
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Moon  Zenith  =  41.00  Moon  Azim  =  268.50  Moon  Brightness 

Rodionces  from  Column  255 
Site:  AF5  Date:  01  Feb  2007  10:00  UTC 


Moon  Zenith  =  41.00  Moon  Azim  =  268.50  Moon  Brightness  =  0.81 

Rodionces  from  Column  255 
Site:  AF5  Date:  01  Feb  2007  10:00  UTC 


Fig.  42.  Model  and  Measured  Radiances,  with  Adaptive  Corr  0.98  for  the  clear  shell  and  1.32  for  the  cloud 
shell  (left);  and  additional  15%  threshold  correction  on  the  clear  shell  and  .90  correction  on  the  cloud  shell 
(right),  for  Site  5,  1  Feb  2007,  1000  GMT.  Thus  the  net  corrections  are  1.13  and  1.19  for  the  clear  and 
cloud  shells  respectively. 


Fig.  43.  Raw  Image  and  Cloud  Algorithm  Results  for  a  case  with  no  moon,  Site  5,  20  June  2007,  0716 
GMT 

Several  more  examples  from  other  time  periods  are  shown  in  the  May  08  power  point 
talk,  which  illustrates  the  results  of  the  Oct  07  interim  moonlight  algorithm  in  comparison 
with  the  May  08  version  that  was  fielded.  This  talk  also  includes  movies  that  are  perhaps 
the  most  persuasive  tool  to  convince  us  that  the  results  are  quite  good  nearly  all  the  time 
(or  alternatively,  the  movies  are  the  best  tool  to  alert  us  if  there  are  problems). 

As  will  be  discussed  in  the  report  for  the  follow-on  contract,  the  threshold  for  optical  fade 
for  the  thin  clouds  appears  to  be  about  a  loss  of  0.7  to  1.0  dB.  The  threshold  for  optical 
fade  for  the  opaque  clouds  appears  to  be  about  a  loss  of  about  8  dB.  Our  sponsors  felt 
that  both  of  these  thresholds  were  very  reasonable,  and  they  were  pleased  with  the  results 
of  the  new  algorithm. 
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Fig.  44.  Model  and  Measured  Radiances,  with  Adaptive  Corr  1.24  for  the  clear  shell  and  0.50  for  the 
opaque  cloud  shell  (left)  and  additional  15%  threshold  correction  on  clear  shell  and  .90  correction  on  cloud 
shell  (right),  for  Site  5,  20  June  2007,  0716  GMT 


9.4.  Site  5  Nighttime  Processing  and  Real  Time  Algorithm 

Site  5  had  been  fielded  prior  to  the  start  of  this  contract,  during  Jan  3 1  -  Feb  2  2007.  One 
of  the  requirements  of  this  contract  was  to  get  the  real  time  algorithm  running  in  the  field. 
Unfortunately,  right  at  the  start  of  the  contract,  the  instrument  was  struck  by  lightning  in 
July  2007.  The  repair  was  completed  January  16,  2008.  The  instrument  ran  well  until 
May  22  2008,  when  the  occultor  failed.  It  was  replaced  on  June  30,  and  also  a  ring  shade 
was  added  at  that  time,  to  shade  the  system  from  lights  on  the  horizon. 

In  order  to  field  the  real  time  algorithm,  it  was  necessary  to  extract  the  algorithm  inputs 
from  the  archival  data.  During  the  interval  of  the  contract  period,  we  also  went  ahead  and 
processed  the  archival  data  that  we  had  available  to  us  at  the  time,  although  much  of  this 
work  was  funded  by  the  follow-on  contract.  We  also  extracted  the  inputs  for  the  2007 
data  at  the  same  time  and  went  ahead  and  processed  it. 

Processing  of  the  archival  data  from  Feb  1  -  July  12  2007  is  documented  in  Memo 
AV09-051t.  Several  examples  are  shown  in  Section  9.3,  and  more  are  shown  in  the 
memo.  The  processing  of  data  from  Jan  17  -  Jun  30  3008  is  documented  in  Memo 
AV09-052t.  The  biggest  change  in  the  inputs  was  required  because  two  quite  bright 
lights  appeared  on  the  horizon,  so  the  inputs  had  to  be  modified  to  adjust  for  the  stray 
light  on  the  dome.  This  adjustment  worked  quite  well,  as  can  be  seen  in  Fig.  45. 

Following  the  repair  of  the  occultor  on  30  June,  the  algorithm  was  adjusted  using  data 
from  30  Jun  -  4  Aug  2008.  The  processing  of  these  data,  as  well  as  the  inputs  used  for 
the  fielded  version  of  the  algorithm,  is  documented  in  Memo  AV09-053t.  A  sample 
result  is  shown  in  Fig.  46. 
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Fig.  45.  Raw  Image  and  Cloud  Algorithm  Results  for  Site  5,  23  January  2008,  0702  GMT 


Fig.  46.  Raw  Image  and  Cloud  Algorithm  Results  for  Site  5,  1  July  2008,  0800  GMT 


The  data  set  documented  in  this  section  has  been  delivered  to  the  sponsor,  except  for  data 
from  Jan  16  -  Feb  21  2008,  which  was  left  out  due  to  an  oversight,  and  delivered  under 
the  next  contract.  The  delivery  is  documented  in  Memo  AV08-038t,  and  the  format  of 
the  delivery  is  documented  in  Memo  AV08-037t.  The  field  version  of  the  night 
algorithm  that  handles  both  starlight  and  moonlight  was  installed  on  August  26  2008,  and 
further  updated  on  September  23  2008.  Data  prior  to  September  23  2008  will  be 
processed  and  delivered  as  archival  data.  Data  starting  on  this  date  may  be  considered 
valid,  until  further  updated.  The  real  time  algorithm  delivery  is  documented  in  AV09- 
024t.  These  three  memos  are  repeated  in  Appendices  1,  2,  and  3  of  this  report. 

We  later  discovered  that  the  cloud  amounts  given  in  the  headers  are  not  correct  for  the 
night  algorithm,  although  the  imagery  is  correct.  We  plan  to  correct  and  redeliver  this 
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archival  data  with  correct  headers  in  the  future,  and  update  the  real  time  code.  The  image 
data  that  was  delivered  to  the  sponsor  should  be  good. 

9.5.  Site  3  Nighttime  Processing  and  Real  Time  Algorithm 

Site  3  was  fielded  under  this  contract  during  April  9  2008.  We  had  problems  with  heat  at 
this  site,  until  a  different  brand  of  cooler  was  installed  on  July  28  2008.  There  were  also 
intermittent  problems  with  the  occultor  during  the  period  of  the  contract.  However,  we 
managed  to  sort  the  archival  data  appropriately,  so  that  any  bad  data  are  identified  as 
“uncertain”.  The  sponsor  was  alerted  not  to  use  data  in  this  category. 

As  with  Site  3,  we  were  only  required  to  deliver  the  field  code  to  run  the  algorithm  in  real 
time,  however  much  of  the  work  required  to  process  the  archival  data  is  the  same. 
Although  we  were  able  to  complete  the  analysis,  and  get  the  real  time  algorithm  in  the 
field  under  this  contract,  the  archival  data  were  not  processed  until  shortly  after  the  time 
of  the  contract.  For  convenience,  we  have  documented  all  of  this  archival  data  in  this 
section. 

When  we  tested  the  algorithm  software  from  Site  5  on  Site  3,  we  found  that  the  site  was 
somewhat  darker,  and  the  Milky  Way  caused  false  identification  of  thin  cloud  in  the 
images.  As  a  result,  we  had  to  add  a  “Milky  Way”  feature  to  the  algorithm.  We  also 
made  sure  that  this  updated  version  would  work  well  at  Site  5. 

Processing  of  the  archival  data  from  Apr  9  -  July  31  2008  is  documented  in  Memo 
AV09-048t.  The  inputs  changed  after  this,  primarily  because  the  installation  of  the  new 
cooler  changed  the  alignment  slightly.  The  processing  of  data  from  Aug  1  -  Sep  1  2008 
is  documented  in  Memo  AV09-049t.  Processing  of  the  Sep  2  -  Oct  28  data  is 
documented  in  Memo  AV09-050t.  These  inputs  were  changed,  because  there  had  been  a 
smudge  on  the  dome  during  the  Aug  1  -  Sep  1  period,  and  it  was  cleaned  up  about  Sep  2. 
This  memo  also  documents  the  inputs  to  the  field  code. 

Sample  results  for  Site  3  are  shown  in  Figs.  47  and  48. 

The  archival  data  set  documented  in  this  section  were  delivered  under  the  follow-on 
contract.  However,  the  field  code  was  delivered  under  this  contract.  The  night  algorithm 
that  handles  both  starlight  and  moonlight  was  installed  on  October  23  2008,  and  further 
updated  on  December  15  2008.  Data  prior  to  this  date  will  be  processed  and  delivered  as 
archival  data.  Data  starting  on  this  date  may  be  considered  valid,  until  further  updated. 
The  real  time  algorithm  delivery  is  documented  in  AV09-024t,  which  is  repeated  in 
Appendix  3  of  this  report. 

We  later  discovered  that  the  cloud  amounts  given  in  the  headers  are  not  correct  for  the 
night  algorithm,  although  the  imagery  is  correct.  We  plan  to  correct  and  redeliver  this 
archival  data  with  correct  headers  in  the  future,  and  update  the  real  time  code.  The  image 
data  that  was  delivered  to  the  sponsor  should  be  good. 
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Fig.  47.  Raw  Image  and  Cloud  Algorithm  Results  for  Site  3,  16  September  2008,  0800  GMT 


Fig.  48.  Raw  Image  and  Cloud  Algorithm  Results  for  Site  3,  2  October  2008,  0600  GMT 


9.6.  Summary  of  Night  Algorithm  Results 

Under  this  contract,  we  made  another  major  step  forward  with  the  night  algorithm,  and 
completed  the  moonlight  portion  of  the  high  resolution  night  algorithm.  This  required 
developing  a  large  amount  of  theory  and  software,  in  order  to  extract  the  shape  of  the 
night  radiance  distribution  under  moonlight,  and  then  use  this  information  in  the 
algorithm.  Visual  evaluation  of  the  results  show  them  to  be  very  good.  Under  the 
follow-on  contract  we  did  more  systematic  evaluation  of  the  accuracy,  and  that  also 
substantiated  this  initial  impression. 

We  were  able  to  set  up  the  algorithm  inputs  for  both  Sites  3  and  5,  and  install  them  both 
in  the  field  running  in  real  time.  In  addition,  we  processed  the  Site  5  night  data  archive. 
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Much  of  the  work  of  setting  up  the  algorithm  inputs  and  processing  the  night  data  was 
partially  funded  by  the  follow-on  contract. 


10.  Summary 

Under  this  contract,  we  finished  refurbishment  of  WSI  Unit  #8,  and  fielded  it  at  Site  3. 
The  WSI  Unit  #4  at  Site  5  had  been  struck  by  lighting,  and  we  received  permission  to 
repair  this  instrument.  As  a  result,  Unit  #4  was  repaired,  and  reinstalled  at  Site  5.  This 
meant  that  we  had  WSI  system  running  at  all  3  main  sites,  Sites  2,  3,  and  5.  Site  7,  the 
SOR  site,  was  also  repaired. 

At  the  start  of  this  contract,  Site  2  was  running  a  real  time  algorithm  in  the  field  that  was 
reasonably  up  to  date,  although  it  did  not  handle  haze  in  the  daytime  well,  nor  did  it 
handle  moonlight  well  at  night.  During  this  contract,  we  completed  development  of  the 
adaptive  feature  of  the  day  algorithm,  and  it  works  very  well  to  handle  variable  haze.  We 
completed  the  moonlight  version  of  the  night  high  resolution  algorithm.  It  is  also 
working  very  well.  We  consider  both  of  these  algorithms  to  be  mature  at  this  point.  We 
recognize  that  there  are  minor  changes  we  might  consider,  but  the  major  changes  that  we 
have  wanted  to  do  for  some  time  are  complete. 

We  put  the  new  algorithms  running  real  time  in  the  field  at  Sites  3  and  5,  for  both  day  and 
night.  We  also  processed  the  archival  data  that  we  had  available  to  us  for  Site  3  2008 
day,  Site  5  2008  day,  and  Site  5  2007  and  2008  night  and  delivered  the  processed  archival 
data  to  the  SOR. 

The  work  was  documented  in  detail  in  memos,  and  in  somewhat  less  detail  in  this  report. 
Much  of  this  is  very  detailed  work,  and  requires  detailed  documentation.  We  apologize 
for  the  volume  of  documentation,  but  this  is  information  we  needed  for  the  future. 

We  were  very  pleased  to  have  all  3  sites  with  operational  WSI  systems  with  algorithms 
running  in  real  time,  and  to  have  begun  on  the  task  of  processing  archival  data. 
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12.2.  Power  Point  Files  from  Presentations  to  SOR 

Note:  Some  of  this  work  was  funded  under  the  follow-on  contract  that  started  in  Feb  08, 
and  some  was  funded  under  the  contract  that  is  the  subject  of  this  report.  We  have  listed 
all  the  talks  given  during  the  interval  of  this  contract,  and  given  a  cryptic  description  of 
the  contents  of  the  talks. 

October  07:  This  trip  was  cancelled  due  to  a  wildfire  near  my  home,  but  the  talk  was 
sent  to  SOR. 

“Whole  Sky  Imager  Update”  provided  hardware  status  and  overview  of  algorithm  status. 
Site  2  was  operating  well,  Site  3  deployment  was  delayed  due  to  camera  repair,  Site  5 
was  hit  by  lightning  and  under  repair,  Site  7  was  brought  back  on-line,  VA  site  was  not 
funded.  Provided  overview  of  concepts  behind  new  moonlight  algorithm,  and  examples 
from  Site  5,  as  well  as  Site  5  starlight  algorithm  results. 

November  07: 

a)  “Whole  Sky  Imager  Current  Cloud  Algorithm  Sample  Results  Developed  at  MPL”. 
Reviewed  night  algorithm  for  starlight,  and  new  results  for  moonlight  with  partially 
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completed  algorithm,  for  Site  5,  and  past  transmittance  results.  Reviewed  day  alg  results 
for  clear  (SOR)  and  hazy  (VA)  site. 

b)  “Using  the  WSI  for  aerosol  characterization”  presented  Alberto  Cazorla’s  results  for 
extracting  aerosol  amount. 

January  08: 

a)  “Whole  Sky  Imager  and  Night  Algorithm  Update”.  Updated  hardware  status:  Site  2 
operating  well,  Site  5  repaired  and  operating  well,  Site  3  nearing  deployment,  SOR  and 
VA  sites  not  funded.  Updated  moonlight  algorithm,  examples  for  Site  5,  presented  blind 
test  and  cloud  assess  results. 

b)  “An  Update  of  Wavelength  Analysis  for  Cloud  Imaging”  analyzed  current  IR  systems, 
SWIR  options,  pros  and  cons. 

c)  “Using  the  WSI  for  aerosol  characterization”  presented  Alberto  Cazorla’s  results  for 
extracting  aerosol  amount.  This  was  a  slightly  updated  version,  presented  to  a  wider 
audience. 

d)  “Whole  Sky  Imager  and  Night  Algorithm  Update  Summary”.  The  above  3  talks  were 
detailed  talks  for  the  working  group.  This  was  the  summary  of  all  of  the  above  for  the 
TIM  group. 

May  08: 

a)  “Whole  Sky  Imager  and  Night  Algorithm  Status  Update”.  Discussed  Site  3 
deployment.  All  3  sites  are  operational.  Provided  operational  statistics.  Site  5  nt  alg 
almost  ready  to  go,  also  working  on  Site  3  nt  alg.  Presented  updated  results  and  stats. 
Showed  transmittance,  and  detennined  threshold  optical  fade. 

b)  “Whole  Sky  Imager  Capabilities  related  to  SWIR  Optical  Fades”.  Reviewed 
extinction  mechanisms,  visible  vs.  SWIR  relationships,  methods  to  extract  optical  fade  at 
night,  several  current  transmittance  results,  range  of  fades  we  can  detect  and  thresholds. 
Presented  method  to  determine  fade  in  daytime. 

c)  “Whole  Sky  Imager  SWIR  Optical  Fades  and  Status  Updates”.  The  above  2  talks 
were  detailed  talks  for  the  working  group.  This  was  the  summary  of  all  of  the  above  for 
the  TIM  group. 

September  08: 

“Whole  Sky  Imager  Status  Update”:  Overview  of  purpose  of  WSI,  and  site  status.  Site  2 
down  due  to  power  surge,  other  sites  operational.  Site  5  night  archival  data  has  been 
processed,  real  time  algs  installed  both  day  and  night,  updated  transmittance  results,  also 
completed  adaptive  alg  for  day.  Discussed  adaptive  alg  and  how  it  works. 
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October  08: 


“Whole  Sky  Imager  Status  Update”:  Hardware  overview.  Sites  3,  5,  7  running  well,  2 
scheduled  for  repair  in  Nov.  Site  5  day  and  night  running  real  time  in  field;  day  2008 
archival  data  processed  and  delivered,  Site  5  night  2007  and  2008  archival  data  processed 
and  delivered.  Site  3  day  and  night  running  real  time  in  field;  day  archival  data  processed 
and  delivered.  Evaluated  available  data  remaining  to  be  processed. 
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Appendix  1:  Contents  of  Memo  AV08-037,  documenting  the  format  of 
the  delivered  data  archive. 

Note:  In  order  to  be  reasonably  clear  within  the  context  of  this  report,  all  of  the  technical 
memo  section  numbers  have  had  “Al.”  added  to  them,  so  that  Section  1  because  Section 
A1 . 1 .  If  the  memo  refers  to  a  given  section  X,  then  it  should  be  understood  that  in  the 
context  of  this  report,  this  becomes  Section  Al.X. 

Subject:  WSI  Cloud  Product  Data  Archive  Format  Overview 

Having  recently  completed  the  night  cloud  algorithm  with  high  resolution  for  both 
starlight  and  moonlight,  as  well  as  the  day  cloud  algorithm  with  an  adaptive  algorithm 
that  corrects  for  haze,  we  are  in  the  process  of  getting  the  required  inputs  for  all  sites,  and 
processing  (or  reprocessing)  the  archival  data.  This  memo  is  intended  to  enable  users  to 
extract  the  cloud  decision  results  from  the  archival  drives  that  we  are  generating.  The 
memo  will  document  the  directory  structure,  file  format,  header  format,  how  to  sort  the 
data  using  the  headers,  the  meaning  of  the  data  levels,  how  to  display  them,  and  the 
approximate  geometry  of  the  image.  The  specific  site,  dates,  geometric  equations,  and 
comments,  will  be  provided  in  separate  memoranda  that  are  associated  with  each  drive 
delivered  to  the  sponsors.  Memo  AV08-038t  documents  the  first  archive  drive. 

Al.l.  Directory  Structure 

The  directory  structure  is  based  on  the  one  that  the  SOR  team  uses  when  they  download 
data  from  the  field.  The  data  will  have  a  directory  name  such  as  “Site  5”.  The  next 
subdirectory  down  will  be  something  descriptive,  such  as 
“Archival2008DaySetlProcl30ct08”.  Under  this  will  be  the  standard  format  used  by 
SOR,  i.e.  the  next  subdirectory  will  be  the  year,  such  as  “2008”.  Below  that  will  be  the 
month  in  the  format  “01”,  and  below  that  will  be  the  day  in  the  format  “01”.  Within  this 
directory  will  be  the  cloud  decision  images  and  a  single  additional  file  for  each  day 
indicating  the  confidence  level  for  each  cloud  decision  image  in  the  directory.  In  the  near 
future,  for  night  images,  we  will  add  transmittance  or  optical  fade  data  files  as  soon  as 
feasible,  although  we  are  putting  priority  on  the  cloud  results  for  the  moment.  Similarly, 
we  anticipate  adding  an  optical  fade  product  for  the  daytime  in  the  longer  tenn. 


A1.2.  File  Format 

The  files  are  named  WSIUUUDDDDYYYYTTTT.cld,  where  UUU  is  unit  number, 
DDDD  is  calendar  day  (month  and  day),  YYYY  is  year,  and  TTTT  is  time  in  GMT. 

The  images  are  512  x  512,  8  bits  deep.  Although  the  raw  data  are  16  bits  deep,  the 
algorithm  results  are  mapped  into  an  8-bit  image,  as  discussed  in  section  5.  Most  of  the 
pixels  represent  algorithm  results,  as  documented  in  Section  5,  although  some  near  the 
top  represent  header  values,  as  documented  in  Section  3. 
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A1.3.  Header  Format 


There  are  9  header  lines  in  the  cloud  decision  images.  The  headers  include  information 
that  enables  us  to  trace  back  to  the  measurement  conditions,  as  well  as  processing 
parameters.  Most  of  the  items  in  the  headers  are  for  internal  use,  however  some  may  be 
useful  to  the  data  users.  We  have  documented  those  parameters  we  feel  may  be  useful. 
In  some  cases,  header  parameters  are  obsolete  and  not  updated,  so  please  do  not  attempt 
to  use  parameters  that  are  not  documented  here. 

This  section  documents  the  headers  for  Units  4,  7,  and  8,  which  are  at  Air  Force  Sites  5, 
2,  and  3  respectively.  Other  systems  may  have  other  fonnats,  and  will  be  documented 
when  we  begin  archival  processing. 

Line  1:  Line  1  documents  the  site,  date  and  time,  as  well  as  various  hardware  readings. 
The  parameters  relevant  to  data  users  are  the  Site,  date,  time,  and  file  (type).  The  other 
parameters  are  for  internal  use.  For  Site,  we  have  used  “AFT”  for  Site  2,  and  “AF3”  and 
“AF5”  for  Sites  3  and  5.  The  SOR  site  currently  has  a  different  header  format.  If  funds 
pennit,  we  hope  to  update  this  in  the  future.  The  file  type  is  given  as  CldNR  or  CldRD 
for  the  day  cloud  decision,  depending  on  whether  it  was  derived  with  Near  Infrared  (NIR) 
or  red  images.  The  night  cloud  decision  images  are  identified  as  “CLR”,  because  the  data 
were  derived  from  the  Clear,  or  Open  hole,  data.  A  sample  of  Line  1  is  shown  below. 

Site: AFT  Bd=  27.9Cew=  82.48  File: CldNR  mage  Day=14 

Month=  9  Year=2006  Time=1615Z  G  Exposure=  100ms  ND=3 

SP=9  Occultor  Destination:  Arc=  70.8  Trle=113.2  Housing 

Temp=26  Hware  Ver:  7.30  Sware  Ver:  1.20  Time  Stat:  4  N2 
pressure=  5  Flow  rate=  0.39  Env.  Housing  Temp=  23  CCD 
Chip  Temp=-29  Occultor  Position:  Arc=  70.6  Trolley=999 . 0 
Rel.  Humidity=  68  Red  flags:  00000000000  Yellow  flags 

0000000000000000 

Line  2:  Line  2  includes  the  sun  and  moon  position,  that  the  user  may  wish  to  use.  The 
user  should  not  use  the  image  position  values  shown  in  the  header;  these  values  are 
estimated  values  before  fielding,  used  in  the  field  acquisition  code,  but  should  not  be  used 
for  the  precise  image  geometry.  (The  image  geometry  will  be  provided  with  each 
archival  drive.)  The  remainder  of  the  line  contains  obsolete  parameters  that  should  not  be 
used.  A  sample  of  Line  2  is  shown  below. 

Sun  Position:  Azimuth=142 . 5  Zenith=  29.8  Moon  Position: 
Azimuth=2 8 9 . 3  Zenith=  62.6  Source=Sun  Azimuth  Offsets: 

Camera=  0.0  Field=  0.0  Occultor  Offsets:  Azimuth=  0.0 
Zenith=  0.0  Image  Center:  Col=254  Row=254  Radius=244 
Field  Azm.  update  time:  0909:09 

Line  3,  4,  and  5:  Lines  3,  4,  and  5  provide  details  of  the  data  acquisition  and  processing 
that  are  used  internally  by  MPL. 
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Line  6:  Parts  of  Line  6  pertain  to  the  processing,  and  other  parts  are  obsolete,  as  they 
refer  to  features  not  used  by  the  SOR  program.  The  obsolete  portions  are  not  checked  for 
accuracy.  However,  the  parameters  starting  at  pixel  418  may  be  useful  to  the  data  user. 
These  parameters  list  the  percent  of  the  full  image  identified  as  clear,  thin,  opaque,  no 
decision,  indeterminate,  and  offscale  bright.  To  determine  clear  fraction,  divide  the  clear 
by  the  sum  of  clear,  thin,  and  opaque.  To  determine  cloud  fraction,  divide  the  sum  of 
thin  and  opaque  by  the  sum  of  clear,  thin,  and  opaque.  The  no  decision  value  is  very 
high,  because  at  the  time  these  instruments  were  fielded,  the  funding  levels  were  very 
low,  and  we  had  to  use  fixed  occultor  shades.  New  occultor  shades  that  are  much  smaller 
are  in  design. 

A  sample  of  Line  6  starting  at  Pixel  418  is  given  below. 

Clear=  4%  Thin=  3%  Opaque=  68%  No  Dec=  24%  Indeter= 
0%  Off  Brt=  0% 

Line  7:  Line  7  is  used  internally  by  MPL. 

Line  8:  Line  8  contains  the  error  strings,  and  bit  18  is  very  important  to  the  user.  Bits  0 
through  11  are  used  to  annotate  abnonnalities  in  data  acquisition.  Bits  12  and  13  are 
saved  positions,  in  case  we  wish  to  add  additional  parameters  in  the  future.  Bits  14-17 
are  blank.  Bit  18  is  the  most  important  to  the  user,  as  it  identifies  images  that  we  feel 
should  not  be  used.  The  decision  was  made  to  go  ahead  and  provide  a  cloud  decision 
even  when  there  is  a  known  error  in  the  raw  data,  but  to  identify  it  in  the  header  as  an 
image  that  is  probably  bad. 

If  bits  0-11  are  all  0’s,  the  quality  value  in  bit  19  will  be  set  to  1.  This  indicates  a  high 
level  of  confidence  in  the  cloud  decision  image.  If  any  of  the  flags  in  the  table  above  are 
set,  the  quality  bit  value  will  be  set  to  2.  This  indicates  that  the  cloud  decision  image 
should  be  examined  because  it  may  be  compromised  due  to  a  system  error.  If  bits  0  or  5 
are  set  to  1,  the  quality  bit  value  will  be  set  to  3.  This  indicates  that  the  cloud  decision 
image  results  are  not  valid  and  should  not  be  included  in  routine  data  analysis. 

The  data  bits  are  listed  below. 

Bit  0  Spectral  filter  error 

Bit  1  Neutral  density  filter  error 

Bit  2  Arc  position  off  by  >=  8  deg. 

Bit  3  Trolley  position  off  by  >=  8  deg. 

Bit  4  Flux  level  -  5%  of  pixels  offscale  bright 

Bit  5  Flux  level  -  offscale  dark 

Bit  6  Flux  level  -  1%  of  pixels  offscale  bright 

Bit  7  Flux  level  -  Possible  shutter  malfunction 

Bit  8  Flow  <  .09 

Bit  9  Env.  Housing  temp  >  49 

Bit  10  Cam.  Housing  temp.  >  49 
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Bit  11  CCD  chip  temp.  >  -1 

Bit  12  Not  used  (always  0) 

Bit  13  Not  used  (always  0) 

Bits  14  -  17  are  blank 

Bit  18  Cloud  Decision  QC  bit 

A  sample  of  Line  8  is  given  below. 

00000000000000  1 

Line  9:  Line  9  is  for  internal  use.  It  exists  in  images  created  after  Sept.6,  2006 
The  remainder  of  the  image  is  non-header  data,  as  described  in  Section  5. 

A1.4.  Sorting  the  Data  for  cases  expected  to  be  valid 

As  noted  earlier,  the  decision  was  made  at  SOR  that  we  should  include  cloud  decision 
images  that  may  not  be  valid  due  to  instrumental  problems,  but  should  identify  them  in 
the  headers  as  being  not  valid.  An  example  might  be  if  the  occultor  hangs,  and  there  is 
stray  light  present  in  the  image. 

There  are  two  ways  for  the  user  to  identify  and  remove  these  images.  The  user  can  read 
the  header,  line  8  pixel  18.  If  the  value  is  3,  this  image  should  not  be  used.  If  the  value  is 
2,  we  will  try  to  advise  you  in  the  specific  memos.  For  example,  if  the  “2”  is  caused  by 
an  arc  error,  as  in  bit  2,  then  the  images  should  not  be  used.  On  the  other  hand,  if  the  “2’ 
is  caused  by  too  many  offscale  bright  pixels,  as  in  bit  6,  that  doesn’t  matter  because  the 
algorithm  will  handle  the  bad  pixels  and  reject  them.  As  we  have  more  opportunity  to 
evaluate  the  processed  data,  we  will  try  to  update  the  criteria  to  make  them  more 
definitive. 

The  second  way  the  user  can  remove  the  images  is  by  use  of  the  CloudDecQCReport.txt 
files.  There  is  a  file  for  each  day,  provided  in  the  subdirectory  for  each  day,  that  provides 
the  result  of  sorting  the  QC  pixel  discussed  above.  A  portion  of  a  sample  output  file  is 
shown  in  Figure  1. 


Figure  1 

A  portion  of  File  CloudDecQCReport.txt 

CloudDecQC  vl.O  Cloud  Decision  Confidence  Level  Report  Page:  1 

Sort  By:  Name  Date:  10/16/08 

Atmospheric  Optics  Group  (UCSD)  Time:  15:35:20 


Directory:  G:\Site  5\Archival  2008  Day  Set  1  Proc  13  Oct  08\2008\01\17\ 


Cloud  File  Name  Confidence  Level 
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WSI00401 1720080000. CLD  1  -  Valid 
WSI0040 1 1 72008000 1  .CLD  1  -  Valid 
WSI00401 1720080002. CLD  1  -  Valid 
WSI00401 1720080003. CLD  1  -  Valid 
WSI00401 1720080004. CLD  1  -  Valid 
WSI00401 1720080005. CLD  1  -  Valid 
WSI00401 1720080006. CLD  1  -  Valid 
WSI00401 1720080007. CLD  1  -  Valid 

At  the  present  time,  times  that  are  not  expected  to  be  valid  due  to  sunrise  or  sunset  are  not 
processed.  Typically,  the  day  is  processed  for  all  Solar  Zenith  angles  (SZA)  of  85 
degrees  or  less.  The  night  data  are  processed  for  all  SZA  values  of  104  or  higher.  We 
will  indicate  what  angles  were  used  in  the  memos  that  go  with  specific  data. 

It  should  be  noted  that  the  QC  checks  are  continuing  to  mature.  If  we  find  problems  that 
we  feel  will  significantly  affect  the  statistics,  but  that  are  not  identified  by  the  QC 
parameters,  we  reserve  the  right  to  remove  these  data  from  the  archival  directory.  In  such 
a  case,  we  will  document  this  decision  in  the  memo  that  goes  along  with  the  delivery. 

A1.5.  Data  format 

The  raw  data  have  been  processed  to  yield  cloud  decisions,  and  the  results  of  the 
decisions  are  coded  into  8  bits  in  the  following  way.  The  color  code  refers  to  the  colors 
used  in  the  displays  that  MPL  produces. 

For  the  day  algorithm: 


Numeric  Range 

Decision 

Color  Code 

0 

No  data 

Black 

1-99 

Clear  or  Haze 

Blue 

100-139 

Thin  cloud 

Yellow 

140-200 

Opaque  cloud 

Grey  to  White 

201 

Offscale  bright 

Bright  White 

202 

Indetenninate 

Dark  Grey 

For  the  night  algorithm: 

Numeric  Range 

Decision 

Color  Code 

0-49 

No  data 

Black 

50-119 

Clear  or  Haze 

Blue 

120-179 

Thin  cloud 

Green 

180-249 

Opaque  cloud 

Grey  to  White 

A  range  of  values  is  used  for  each  category  such  as  "opaque",  such  that  we  can  retain 
characteristics  of  the  original  image  for  ease  in  viewing.  For  example,  once  a  pixel  is 
identified  as  day  opaque  cloud,  its  value  is  assigned  within  the  range  of  140  -  200  based 
on  the  relationship  between  the  pixel's  actual  ratio  and  the  threshold  ratio.  At  the  present 
time,  variations  within  each  category  are  not  technically  meaningful,  but  are  only  used  in 
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creating  the  display  image.  The  color  look-up  tables  for  displaying  the  images  are  given 
in  Appendices  1  and  2.  These  color  tables  are  for  the  version  that  shows  solid  blue  in  the 
“clear  or  haze”  category. 

For  night,  the  threshold  between  “clear  or  haze”  and  “thin  cloud”  is  approximately  0.8 
dB.  The  threshold  between  “thin  cloud”  and  “opaque  cloud”  is  approximately  8  dB.  For 
daytime,  we  have  not  yet  extracted  these  thresholds;  however  the  same  values  are  good 
working  numbers.  That  is,  we  expect  the  thresholds  to  be  similar,  because  the 
appearance  of  the  thin  clouds  that  are  missed  by  the  night  and  the  day  algorithms  are 
similar.  As  mentioned  earlier,  we  plan  to  add  the  transmittance  or  optical  fade  table  data 
product  at  night  as  soon  as  feasible,  and  plan  to  follow  up  with  work  to  provide  optical 
fade  for  the  daytime. 

A1.6.  Image  Geometry 

Because  the  images  are  acquired  looking  up,  they  do  not  have  the  same  directions  as 
looking  down  on  a  map.  Specifically,  North  is  at  the  bottom  of  the  image,  South  at  the 
top,  East  on  the  right  edge,  and  West  on  the  left.  The  image  typically  has  about  a  181.5 
degree  field  of  view,  and  each  pixel  is  about  1/3  degree.  The  specific  equations  relating 
pixel  location  to  angular  location  and  visa  versa  will  be  documented  with  each  data  set. 

A1.7.  Upgrades 

There  may  be  additional  upgrades  to  the  algorithms  in  the  future.  As  an  example, 
occasionally  a  very  small  rim  of  opaque  cloud  is  identified  near  the  aureole.  We  chose  to 
get  this  data  out  now,  rather  than  take  the  time  to  fix  it.  But  if  it  causes  problems  with  the 
data  analysis,  we  would  plan  to  fix  it  and  reissue  the  archival  data.  For  this  reason,  if  the 
users  find  problems  with  these  data  sets,  we  appreciate  being  politely  alerted  so  that  we 
can  make  the  necessary  upgrades. 

Sub-Appendix  1.1:  Code  for  Color  Table  for  Day  Algorithm  Colors 

/*  -  DECIDE  function  VgaViewDecision  -- 

-  */ 

void  SetCldPalette  (imgdes  *image) 


{  int  j  ,  j  t3  ; 

/*  Set  the  cloud  decision  color  palette  -  First  set  up  gray¬ 
scale  */ 

image- >palette [0] . rgbRed  =0;  /*  Set  no  data 

(0)  to  black  {0,0,0}*/ 

image- >palette [0] . rgbGreen  =  0; 
image- >palette [0] . rgbBlue  =  0; 


/*  Set  the  clear  sky  range.  Make  it  solid  blue  */ 
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for  (j  =  1  ;  j  <  100  ;  j++) 

image->palette [ j ] . rgbRed  =  90; 
image->palette [ j ] . rgbGreen  =  90; 
image->palette [ j ] . rgbBlue  =  194; 

} 


/*  Set  the  thin  cloud  range  */ 

for  (j  =  100  ;  j  <  140  ;  j++)  /*  make  it  yellow  */ 

{  jt3  =  j  *  3; 

image->palette [ j ] .rgbRed  =  (UCHAR)  20  +  (UCHAR)  (  (float 
)  1.3*  (float  )  j 

+  (float  )  .5)  ;  /*  Red 

value  */ 

image- >palette [j ] .rgbGreen  =  image- 
>palette[j] .rgbRed;  /*  Green  value  */ 

image- >palette [j ] .rgbBlue  =0;  /*  Blue  value 

*/ 

} 


/*  Set  the  opaque  cloud  range  */ 

for  (j  =  14  0  ;  j  <  2  01  ;  j++) 

{  jt3  =  j  *  3; 

image->palette [ j ] .rgbRed  =  (UCHAR)  68  +  (UCHAR)  (  (float 
)  .  8  *  (float  )  j 

+  (float  )  .  5 )  ; 

/*  Red  value  */ 

image- >palette [j ] .rgbGreen  =  image- 
>palette[j] .rgbRed;  /*  Green  value  */ 

image- >palette [j ] .rgbBlue  =  image- 
>palette[j] .rgbRed;  /*  Blue  value  */ 

} 


image- >palette [201] .rgbRed  =  240;  /*  Set  off  scale  (201) 

to  {240,240,240}  */ 

image- >palette [201] .rgbGreen  =  240; 
image- >palette [201] .rgbBlue  =  240; 


image- >palette [202] .rgbRed  =  40;  /*  Set  indeterminate 

(202)  to  {40,40,40}  */ 

image- >palette [202] .rgbGreen  =  40; 
image- >palette [202] .rgbBlue  =  40; 


for  (j  =  203 
(UCHAR)  (j); 

for  (j  =  203 
(UCHAR)  (j); 

for  (j  =  203 
(UCHAR)  (j); 


j  <  256  ;  j++) 
j  <  256  ;  j++) 
j  <  256  ;  j++) 


image- >palette [j ] .rgbRed  = 
image- >palette [j ] .rgbGreen  = 
image- >palette [j ] .rgbBlue  = 
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}  /*  End  function  SetCldPalette  */ 

Sub-Appendix  1.2:  Code  for  Color  Table  for  Night  Algorithm  Colors 

void  SetNightCldPalette  (imgdes  *image) 


{  int  j  ,  j  t3  ; 

/*  Set  the  cloud  decision  color  palette  -  First  set  up  gray¬ 
scale  */ 

for  (j  =  0;  j  <  50;  j++) 

{ 

image- >palette [j ] . rgbRed  =0;  /*  Set 

no  data  (0)  to  black  {0,0,0}*/ 

image- >palette [j ] . rgbGreen  =  0; 
image- >palette [j ] . rgbBlue  =  0; 

} 


/*  Set  the  clear  sky  range  */ 

for  (j  =  5  0  ;  j  <  12  0  ;  j+  +  ) 

{ 

//Below  is  for  varying  shades  of  blue 
// j  t3  =  j  *3; 

//image- >palette [j ]. rgbRed  =  (unsigned  int)  (  ( j - 5 0 )  * 
100. /70.)  +  60;  /*  Red  value  */ 

//image- >palette [j ]. rgbGreen  =  image- 
>palette [j ]. rgbRed;  /*  Green  value  */ 

//image- >palette [j ]. rgbBlue  =  (unsigned  int)  ( ( j  -  50)  * 
100. /70.)  +  150;  /*  Blue  value  */ 

//Version  with  solid  blue 

image->palette [ j ] .rgbRed  =  (  unsigned  int) 

60;  /*  Red  value  */ 

image->palette [ j ] .rgbGreen  =  image- 
>palette[j] .rgbRed;  /*  Green  value  */ 

image->palette [ j ] .rgbBlue  =  150;  /*  Blue 

value  */ 

} 


/*  Set  the  thin  cloud  range.  Make  it  green*/ 

for  (j  =  120  ;  j  <  180  ;  j++) 

{  jt3  =  j  *  3; 


image->palette [ j ] 
70. /60.)  +  80; 

image->palette [ j ] 
*  70 . /6 0  .  )  +  150;  ; 

image->palette [ j ] 
>palette[j] .rgbRed; 

} 


.rgbRed  =  (  unsigned  int  ) 

/*  Red  value  */ 

.rgbGreen  =  (  unsigned  int 

/*  Green  value  */ 
.rgbBlue  =  image - 

/*  Blue  value  */ 


((j  - 
)  ((j 


120)  * 
-  120) 
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/*  Set  the  opaque  cloud  range  */ 

for  (j  =  180  ;  j  <  250  ;  j++) 

{  j  1 3  =  j  *  3  ; 

//Oldimage- >palette [ j ] . rgbRed  =  (unsigned  int)  ( (j  - 
180)  *  170. /IQ.)  +  80; 

image->palette [ j  ]  .rgbRed  =  (  unsigned  int)  (  ( j  -  180)  * 
50. /70.)  +  180;  //New  per  JS  22Sep06  Red 

value  */ 

image->palette [ j ] . rgbGreen  =  image- 
>palette[j] .rgbRed;  /*  Green  value  */ 

image->palette [ j ] . rgbBlue  =  image- 
>palette[j] .rgbRed;  /*  Blue  value  */ 

} 


}  /*  End  function  SetNightCldPalette  */ 
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Appendix  2:  Contents  of  Memo  AV08-038,  documenting  the  delivered 
data  archive. 

Note:  In  order  to  be  reasonably  clear  within  the  context  of  this  report,  all  of  the  technical 
memo  section  numbers  have  had  “A2.”  added  to  them,  so  that  Section  1  because  Section 
A2.1.  If  the  memo  refers  to  a  given  section  X,  then  it  should  be  understood  that  in  the 
context  of  this  report,  this  becomes  Section  A2.X. 

Subject:  Delivery  of  WSI  Cloud  Data  Archival  External  Drive  Serial  Number 
L80QDGFG 

We  have  prepared  a  drive  with  archival  cloud  algorithm  data,  for  hand  delivery  to  SOR 
on  23  October  2008.  Memo  AV08-037t  provides  an  overview  of  the  data  fonnat.  It  is 
important  to  either  sort  the  data  as  noted  in  Section  4  of  that  memo,  or  use  the 
CloudDecQCReport.txt  file  that  is  included  in  each  subdirectory  and  documented  in 
Section  4  of  that  memo.  This  file  provides  a  summary  for  each  day  of  the  cases  in  which 
the  raw  data  are  abnormal  and  the  cloud  decision  data  are  not  expected  to  be  valid.  As 
stated  in  Section  4  of  Memo  AV08-037t,  the  cloud  decision  images  in  the  file 
CloudDecQCReport.txt  are  labeled  as  ‘Valid’,  ‘Uncertain’  or  ‘Invalid’.  The  invalid  cases 
should  not  be  used.  The  uncertain  cases  also  should  normally  not  be  used.  In  the  future, 
we  plan  to  evaluate  the  data  in  more  detail  and  can  reassess  this,  as  in  some  cases  it  may 
be  Ok  to  use  the  uncertain  cases. 

This  memo  documents  the  specifics  of  what  is  included  in  this  delivery.  The  External 
Drive  has  serial  number  L80QDGFG.  Because  our  sponsors  are  asking  for  data  as 
quickly  as  possible,  we  opted  to  send  what  we  could  readily  process.  In  each  section,  we 
discuss  what  was  and  was  not  processed,  and  why.  The  additional  data  that  has  not  been 
processed  will  be  sent  as  quickly  as  we  can. 

I  would  also  like  to  note  that  we  have  visually  reviewed  the  hourly  images,  but  we  have 
not  run  a  program  such  as  a  blind  test  to  assess  the  overall  accuracy.  We  believe  the  day 
data  are  coming  out  better  than  the  sites  we  assessed  before,  which  had  an  overall 
accuracy  of  over  97%.  Similarly,  the  night  data  appear  to  be  coming  out  better  than  the 
data  we  assessed  before,  which  had  an  overall  accuracy  of  about  90-95%.  Of  this  error, 
we  believe  most  of  it  was  actually  thin  cloud  that  fell  below  the  thin  cloud  threshold  of 
about  0.8  dB,  and  not  actual  error.  We  plan  to  assess  the  accuracy  of  the  data  sets  on  this 
drive  in  the  future. 

A2.1.  Site  5  Day  2008  Set  1  Archive 

This  section  documents  the  Site  5  Day  2008  Set  1  data. 

A2.1.1  Directory  Name  on  Archival  Drive:  Site5\Archival2008DaySetlProcl30ct08. 
The  date  indicates  the  date  that  processing  started. 

A2.1.2  Dates  Processed:  Jan  16  2008  -  May  20  2008 
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A2.1.3  Solar  Zenith  Angles  Processed:  0-85 


A2.1.4  Algorithm  Used:  These  data  used  the  first  version  of  the  adaptive  algorithm. 
Program  AutoProc2.4-20. 

A2.1.5  General  Comments: 

The  data  in  general  look  quite  good.  In  some  instances,  the  solar  aureole  has  edges 
identified  as  opaque.  We  know  how  to  fix  this,  but  did  not  want  to  slow  down  to  do  this 
at  this  time.  Also,  we  may  make  further  upgrades  to  the  adaptive  algorithm  in  the  future, 
as  there  are  occasional  times  when  it  misses  more  cirrus  than  we  would  like.  If  it  does 
miss  cirrus,  it  normally  identifies  some  of  the  cirrus  in  the  image,  and  we  believe  the 
problem  occurs  in  about  2%  of  the  images.  We  feel  that  the  clear  cases  and  the  opaque 
cases  are  normally  identified  correctly. 

A2.1.6  Processing  of  Day  Site  5  Data: 

A  record  of  the  data  availability  and  data  processed  for  Site  5  is  shown  in  the  Appendix. 

The  instrument  was  deployed  during  3 1  Jan  -  1  Feb  07.  It  ran  until  struck  by  lightning  in 
July  07.  The  repair  was  completed  Jan  16  2008.  The  occultor  ACP  failed  May  22  2008, 
and  was  repaired  June  30  -  July  2  2008.  The  2007  daytime  data  have  not  yet  been 
processed,  because  we  will  have  to  probably  pull  a  new  clear  sky  background  for  that  set. 

The  daytime  data  for  Set  1  2008,  16  Jan  -  20  May  08,  are  processed  and  delivered  with 
this  delivery.  The  processing  computer  stopped  during  the  run  (perhaps  due  to  a  power 
outage),  so  we  will  have  to  separately  run  the  21-22  May  08  data.  The  daytime  data  for 
Set  2  2008  are  the  data  since  July  2.  We  received  Set  2  test  data  August  12,  and  the  real 
time  algorithm,  tested  with  Set  2  data,  was  installed  August  26.  An  upgrade  to  the 
algorithm  was  installed  14  October.  The  data  since  August  26  should  be  valid,  but  we 
will  process  all  data  from  July  2  through  October  14  after  we  receive  the  data.  The  night 
data  for  Site  5  has  been  processed,  and  will  be  discussed  below. 

A2.1.7  Equations  to  relate  Pixel  Position  to  Angle: 

The  following  equations  may  be  used  to  convert  between  image  coordinates  and  spherical 
coordinates.  The  zenith  and  azimuth  are  in  degrees,  and  azimuth  is  clockwise  with 
respect  to  True  North  in  real  space  (not  in  image  space).  That  is,  the  bottom  of  the  image 
is  North,  and  the  right  side  is  East,  as  explained  in  Memo  AV08-037t. 

The  equations  for  azimuth,  (j),  and  zenith,  9  in  degrees,  given  pixel  location  x,y  are: 

(j)  =  0.05787  +  fa  -  0.004426  pCo^Tifa  / 90)  +  0.0002416  p  Sxxfanfa  /  90) 


(1) 
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(2) 


d  =  0.3436  p  -  0.00004412 p 2  +  7. 145  x  10‘7  /?3  +  0.0006305  p  Cos(^ (/)!  90)  - 
0.004436/?  Sin(^/ 90) 


where  <f>0  =  arctan 


(v- v  ) 

center  / . 


and  p  = 


x  -x„ 


•)2  +(v-JCCTte-)2 


The  geometric  calibration  is  affected  when  the  instrument  is  moved,  such  as  during 
repairs.  In  this  case,  the  image  center  changed,  but  camera  rotation  was  minimal. 
Therefore,  three  different  sets  of  image  center  values  are  valid  for  the  above  equations. 
For  data  through  July  2007,  xcenter=  244.3  and  ycenter=  250.5.  For  data  from  Jan-Jun 

2008,  xcenter  =  246.3  and  ycenter  =  239.5.  Finally,  for  data  from  July  2008  and  later,  xcenter  = 
243.8  and  ycenter=  240.5. 


To  derive  the  x,y  pixel  position  corresponding  to  a  given  angular  position  0,  m,  in 


degrees,  the  equations  are: 

X  =  XC  +PxSin(mf>P/m) 

Y  =  YC  +  Px  Cos{K(f)P  / 1 80) 

where  P  and  (j)p  are  defined  by 

P  =  2.932  6  -  0.0002749  61  -  0.00003042  £3 
0.03171  #Sin(;r0/9O) 


(3) 

(4) 

0.004269  6  Cos(;r0/9O)  +  (5) 


<j>p  =  (j)  -  (0.05787  -  0.004426  P  Cos(;r^/90)  +  0.0002416  P  Sin(^^/90)) 


(6) 


A2.2.  Site  3  Day  2008  Archive 

This  section  documents  the  Site  3  Day  2008  Set  1  data. 

A2.2.1  Directory  Name  on  Archival  Drive:  Site3\Archival2008DaySetlProcl0Oct08. 
The  date  indicates  the  date  that  processing  started. 

A2.2.2  Dates  Processed:  Apr.  9  2008  -  Aug.  4  2008 

A2.2.3  Solar  Zenith  Angles  Processed:  0-85 

A2.2.4  Algorithm  Used:  These  data  used  the  first  version  of  the  adaptive  algorithm, 
Program  AutoProc2.2-20. 

A2.2.5  General  Comments: 
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This  site  had  more  start-up  hardware  problems  than  normal  due  to  the  extreme  heat.  At 
times  the  outside  temperature  was  over  120  F,  and  under  these  conditions  both  the 
camera  electronics  unit  and  the  occultor  brake  had  difficulties.  Although  the  QC 
indicators  are  able  to  catch  some  of  the  bad  cases,  it  was  not  sophisticated  enough  to 
catch  others.  For  this  reason,  we  have  deleted  those  days  that  we  feel  are  adversely 
impacted.  It  is  possible  that  we  may  be  able  to  develop  more  sophistication  in  either  the 
algorithm  (to  identify  the  bad  pixels)  or  the  QC  (to  identify  the  bad  images)  in  the  future, 
in  which  case  we  would  issue  the  data  at  that  time. 

The  cooler  was  replaced  on  July  29  2008,  and  this  solved  the  camera  problems.  The 
dates  that  have  been  deleted  are  April  17-22  and  25,  May  10  -  12,  16,  and  18-21,  June 
24  -  30,  and  July  1  -  28.  In  addition,  the  raw  data  from  April  26  through  May  8  was  lost 
due  to  a  combination  of  failed  ftp,  and  an  external  drive  damaged  in  transit.  The  WSI 
will  save  the  data  if  ftp  is  down  and  send  it  later,  but  if  the  WSI  sends  the  data  and  it  is 
damaged,  the  external  drive  is  normally  used  to  recover  the  lost  data.  May  9  was  also 
deleted,  because  there  were  people  working  on  the  WSI.  At  this  site,  the  “uncertain” 
cases  are  due  to  the  occultor  being  off,  and  should  not  be  used.  (Removing  the  data  as 
noted  above  may  have  removed  all  these  cases.) 

For  those  data  that  were  not  affected  by  the  heat,  the  cloud  results  look  very  good.  Most 
clear  days  are  detennined  correctly  and  when  they  are  not,  it  is  only  at  sunset  where  there 
is  a  slight  undefined  region.  Most  opaque  and  thin  clouds  are  identified  correctly.  On 
some  partly  cloudy  days,  there  are  regions  near  the  clouds  that  probably  have  too  much 
thin  cloud  identified,  although  these  regions  may  also  be  caused  by  cloud  debris. 

A2.2.6  Processing  of  Day  Site  3  Data: 

A  record  of  the  data  availability  and  data  processed  for  Site  3  is  shown  in  the  Appendix. 

The  instrument  was  deployed  during  8-10  Apr  2008.  As  noted  above,  some  periods 
have  problems  due  to  the  heat,  and  a  short  period  of  data  was  lost  during  Apr  26  -  May  8 
due  to  a  problem  with  the  ftp  and  external  drive.  Those  dates  strongly  affected  by  the 
heat  problems  were  deleted  from  the  archive.  The  real  time  algorithm  was  installed 
October  14.  At  the  present  time,  we  have  processed  all  the  data  we  have  received,  which 
is  the  data  from  April  9  through  August  4  2008.  We  will  process  the  data  between 
August  4  and  October  14  after  we  receive  the  data  from  the  field,  and  will  deliver  it  in  a 
later  delivery.  Similarly,  the  night  data  for  site  3  will  be  processed  in  a  later  delivery. 

A2.2.7  Equations  to  relate  Pixel  Position  to  Angle: 

The  instrument  was  moved  slightly  during  replacement  of  the  environmental  housing  that 
was  completed  on  30  July  2008.  The  first  set  of  equations  is  used  for  data  collected 
through  July  2008. 

The  equations  for  azimuth,  (j),  and  zenith,  9  in  degrees,  given  pixel  location  x,y  are: 
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0=-O.8812+0o  +0.0004301/?Cos(^0/90)  +  0.007124/?Sin(^0/90) 


(V) 


0=  0.3365  /?  + 2.322  xlO'V  +  3. 818x  10'7 /?3  -  0.006826/?  Cos(;r0/9O)  -  (8) 

0.0001 1 09  /?Sin(;r0/  90) 

where  0O  =  arctan  ^_Xcewfe,\  and  p  =  ^{x- xcenterf  +  {y  -  ycenterf  .  For  these 

_  IT  y  center  )  _ 

equations,  xcenter  =  249 .7  and  v(;en/er  =  251.7. 

To  derive  the  x,y  pixel  position  corresponding  to  a  given  angular  position  0,  0,  in 


degrees,  the  equations  are: 

X  =  Xc  +  P  x  Sin{n(j)P  / 1 80)  (9) 

Y  =  Yc+PxCos(a<f>P/ 180)  (10) 

where  P  and  (/)p  are  defined  by 

P  =  2.980  6  -  0.001206  61  -  0.00001754  #3  +  0.0521 1 9  Cos(;r0/9O)  +  (11) 

0.0008375  6  Sin(;r0/9O) 

0p  =0  +  0.8812  -0.0004301  P  Cos(;r  0/90)  -0.007124  P  Sin(;r0/9O))  (12) 


The  following  set  of  equations  is  used  for  data  collected  after  July  2008. 

The  equations  for  azimuth,  0,  and  zenith,  0  in  degrees,  given  pixel  location  x,y  are: 

0=  0.6730 +  0O  -0.0005058  /?Cos(/r0o  / 90)  +  0.007731  p  Sin(/r0o /90)  (13) 

0=  0.3366/7  +  1.179x10' 5 p2  +4.242xl0'7  pi  -  0.006667/?  Cos(;r  0/90)  -  (14) 

0.0005 109/?  Sin(;r  0/90) 

where  0O  =  arctan  and  p  =  ^{x-  xcenter  )2  +  (y  -  ycenter  )2  .  For  these 

_  (T  y  center  /  _ 

equations,  xcenter=  249.9  and y center  =  250.5. 

To  derive  the  x,y  pixel  position  corresponding  to  a  given  angular  position  0,  0,  in 
degrees,  the  equations  are: 

X  =  Xc  +  Px  Sin{7uj>P  / 1 80)  (15) 

Y  =  Yc  +Px  Cos{ntj)p  / 1 80)  (16) 

where  P  and  (j)p  are  defined  by 
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(17) 


P  =  2.980  0-  0.0009243  62  -  0.00002075  Oi  +0.051 19  £  Cos(^/90)  + 

0.003791  £Sin(;r^/90) 
q \  =(f)-  0.6730  +  0.0005058  P  Cos(;r  0/  90)  -  0.00773 1 P  Sin(;r  0/  90))  (18) 

A2.3.  Site  5  Night  2007  Archive 

This  section  documents  the  Site  5  Night  2007  data. 

A2.3.1  Directory  Name  on  Archival  Drive:  Site5\Archival2007NightSetProc8Jul08. 

The  date  indicates  the  date  that  processing  started. 

A2.3.2  Dates  Processed:  Feb.  1  2007  -  Jul.  12  2007 

A2.3.3  Solar  Zenith  Angles  Processed:  104  and  greater 

A2.3.4  Moon  Zenith  Angles  Processed:  0  and  greater. 

A2.3.5  Algorithm  Used:  These  data  used  the  new  moonlight  night  cloud  decision 
algorithm  with  ProcWSID  Version  2.4.  This  version  is  the  first  one  that  is  good  for  both 
starlight  and  moonlight. 

A2.3.6  General  Comments: 

The  night  cloud  detection  software  performed  well  for  no-moon  and  moonlight 
conditions.  Tests  based  on  earlier  runs  that  were  not  yet  quite  optimized  showed  90  - 
95%  accuracy  for  zenith  angles  <  60°.  Very  thin  cloud  gets  missed  in  some  cases 
(identified  as  cloud-free),  but  this  issue  may  be  reduced  with  future  updates  to  nominal 
background  imagery. 

A2.3.7  Processing  of  Night  Site  5  2007  Data: 

A  record  of  the  data  availability  and  data  processed  for  Site  5  is  shown  in  the  Appendix. 

The  instrument  was  deployed  during  3 1  Jan  -  1  Feb  07.  It  ran  until  struck  by  lightning  in 
July  07.  All  of  this  2007  data  have  been  processed  through  the  night  algorithm  and  are 
included  on  the  drive. 

A2.3.8  Equations  to  relate  Pixel  Position  to  Angle: 

The  same  equations  given  in  Section  1  of  this  memo  may  be  used  for  the  night  data, 
being  careful  to  use  the  image  center  for  the  correct  period. 

A2.4.  Site  5  Night  2008  Set  1  Archive  (data  through  Jun.  30,  2008) 
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This  section  documents  the  Site  5  Night  2008  Set  1  data. 

A2.4.1  Directory  Name  on  Archival  Drive:  Site5\Archival2008NightSetlProc31Jul08. 
The  date  indicates  the  date  that  processing  started. 

A2.4.2  Dates  Processed:  Feb.  21  2008  -  Jun.  30  2008 

A2.4.3  Solar  Zenith  Angles  Processed:  104  and  greater 

A2.4.4  Moon  Zenith  Angles  Processed:  0  and  greater. 

A2.4.5  Algorithm  Used:  These  data  used  the  new  moonlight  cloud  decision  algorithm 
with  ProcWSID  Version  2.4 

A2.4.6  General  Comments: 

Nighttime  data  collected  during  this  period  were  occasionally  affected  by  image  artifacts 
and  offscale  pixels  caused  by  multiple  light  sources  added  to  the  site  prior  to  camera 
repairs.  New  inputs  were  created  to  account  for  a  shift  in  camera  position,  re-calibration 
of  the  camera,  and  to  account  for  stray  light  effects.  Though  no  test  has  been  performed, 
results  appear  to  be  comparable  to  2007  results.  Also,  during  May  23  -  June  30,  some 
data  are  marked  with  a  QC  level  of  2,  or  “uncertain”,  during  the  May  23  -  June  30 
period.  This  is  because  the  occultor  had  failed,  so  starlight  data  are  good,  but  some 
moonlight  data  are  not  valid  because  the  occultor  was  not  in  place.  These  flagged  data 
should  not  be  used. 

A2.4.7  Processing  of  Night  Site  5  2008  Set  1  Data: 

A  record  of  the  data  availability  and  data  processed  for  Site  5  is  shown  in  the  Appendix. 

In  2008,  the  repaired  instrument  was  installed  Jan  16  2008.  The  occultor  ACP  failed 
May  22  2008,  and  was  repaired  and  a  new  shade  was  installed  June  30  -  July  2  2008. 
Set  1  consists  of  the  data  from  Jan  16  -  June  30  2008.  During  May  23  -  June  30,  only 
the  starlight  data  are  valid,  but  the  moonlight  data  are  flagged  as  “uncertain”  and  should 
be  removed.  Of  this  period,  all  but  Jan  16  -  Feb  20  have  been  processed.  This  residual 
data  will  be  processed  and  delivered  with  the  next  delivery. 

A2.4.8  Equations  to  relate  Pixel  Position  to  Angle: 

The  same  equations  given  in  Section  1  of  this  memo  may  be  used  for  the  night  data, 
being  careful  to  use  the  image  center  for  the  correct  period. 

A2.5.  Site  5  Night  2008  Set  2  Archive  (data  from  Jun.  30,  2008  -  Aug.  4,  2008) 

This  section  documents  the  Site  5  Night  2008  Set  2  data. 
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A2.5.1  Directory  Name  on  Archival  Drive: 

Site5\Archival2008NightSet2Procl60ct08.  The  date  indicates  the  date  that  processing 
started. 

A2.5.2  Dates  Processed:  Jul.  1  2008  -  Aug.  4  2008 
A2.5.3  Solar  Zenith  Angles  Processed:  104  and  greater 
A2.5.4  Moon  Zenith  Angles  Processed:  0  and  greater. 

A2.5.5  Algorithm  Used:  These  data  used  the  new  moonlight  cloud  decision  algorithm 
with  ProcWSID  Version  3.1. 

A2.5.6  General  Comments: 

A  light  shade  was  added  to  the  camera,  which  eliminated  most  of  the  stray  light  artifacts 
found  in  the  earlier  2008  nighttime  data.  New  inputs  were  created  to  account  for  the 
shade,  shift  in  camera  position,  and  elimination  of  the  stray  light  issue.  As  with  earlier 
2008  nighttime  data,  initial  assessment  of  the  results  indicate  accuracy  is  on  par  with 
2007  test  results. 

A2.5.7  Processing  of  Night  Site  5  2008  Set  2  Data: 

A  record  of  the  data  availability  and  data  processed  for  Site  5  is  shown  in  the  Appendix. 

The  instrument  was  repaired  and  a  new  shade  was  installed  June  30  -  July  2  2008.  We 
received  Set  2  test  data  August  12,  and  the  real  time  algorithm,  tested  with  Set  2  data, 
was  installed  August  26.  An  upgrade  to  the  algorithm  was  installed  23  September.  We 
have  processed  the  data  from  July  1  through  August  4  2008.  The  data  since  August  26 
should  be  valid,  but  we  will  process  all  data  from  August  5  through  September  23  after 
we  receive  the  data. 

A2.5.8  Equations  to  relate  Pixel  Position  to  Angle: 

The  same  equations  given  in  Section  1  of  this  memo  may  be  used  for  the  night  data, 
being  careful  to  use  the  image  center  for  the  correct  period. 

A2.6.  Next  Delivery 

At  the  present  time,  we  are  working  on  Site  3  Night  data,  as  well  as  Site  2  daytime  data. 
The  Site  2  data  is  especially  important,  because  it  includes  a  long  data  archive  starting  in 
2006.  We  also  plan  to  include  missing  pieces  from  this  data  set,  such  as  the  Site  5  May 
21  and  22  2008  daytime  data  mentioned  in  Section  1.6. 
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Sub- Appendix  2.1:  Current  Data  Availability  for  Site  5  and  3 
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Appendix  3:  Contents  of  Memo  AV09-024,  documenting  the  use  of  the 
real  time  algorithms. 

Note:  In  order  to  be  reasonably  clear  within  the  context  of  this  report,  all  of  the  technical 
memo  section  numbers  have  had  “A3.”  added  to  them,  so  that  Section  1  because  Section 
A3. 1 .  If  the  memo  refers  to  a  given  section  X,  then  it  should  be  understood  that  in  the 
context  of  this  report,  this  becomes  Section  A3.X. 

Subject:  Delivery  of  Real  Time  Cloud  Algorithms 

At  the  time  that  we  delivered  the  first  archival  cloud  algorithm  data  to  SOR,  we  also 
delivered  Memos  AV08-037t  that  provides  an  overview  of  the  data  format,  and  Memo 
AV08-038t,  that  documents  the  actual  contents  of  the  archival  delivery,  as  well  as  the 
specific  geometric  calibrations  for  that  data.  At  that  time,  our  sponsors  asked  if  we  could 
create  a  memo  similar  to  AV08-038t  that  documents  the  delivery  of  the  cloud  algorithms 
for  real  time  processing. 

Monette  recently  wrote  Memo  AV09-005t  that  documents  the  actual  software  versions 
and  when  they  were  fielded,  as  well  as  a  brief  statement  on  their  differences.  In  this 
memo,  we  will  document  the  additional  infonnation  needed  to  interpret  the  cloud  data 
processed  in  the  field.  We  have  also  recently  completed  a  second  archival  of  processed 
data,  and  that  data  is  documented  in  Memo  AV09-023t. 

A3.1.  Site  2  Day  Real  Time  Cloud  Algorithm 

This  section  documents  the  Site  2  Day  real  time  cloud  algorithm  status. 

A3. 1.1  Overview  of  Earlier  Deliveries:  The  first  version  of  the  day  algorithm  set  up 
with  the  inputs  extracted  for  Site  2  was  installed  on  Sep  6  2006.  The  day  algorithm 
extraction  is  documented  in  Memo  AV06-020t.  More  recently,  the  day  algorithm  was 
upgraded  to  include  the  adaptive  algorithm,  to  better  handle  haze,  and  the  inputs  have 
been  updated.  The  rest  of  this  section  documents  this  version  of  the  code. 

A3. 1.2  Date  Installed:  The  real  time  algorithm  with  the  new  adaptive  algorithm  was 
sent  to  SOR  on  Feb  12  2009.  They  will  install  it  on  the  system  as  soon  as  practical.  This 
has  been  delayed  because  there  is  a  problem  with  the  link  from  SOR  to  the  WSI 
processing  computer.  Data  prior  to  this  date  will  be  processed  and  delivered  as  archival 
data.  Data  starting  on  this  date  may  be  considered  valid,  until  further  updated. 

A3. 1.3  Solar  Zenith  Angles  Processed:  0-85 

A3. 1.4  Algorithm  Used:  This  algorithm  is  ProcWSID  Version  3.5,  which  will  be 
documented  in  a  future  memo. 

A3. 1.5  General  Comments: 
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These  algorithm  inputs  are  extracted  using  the  data  from  Jan  1  -  Jul  3 1  2008  documented 
in  Memo  AV09-007t.  Minor  updates  to  the  algorithm  are  based  on  the  data  from 
November  14,  2008  to  January  27,  2009  and  are  documented  in  Memo  AV09-019t. 

A3. 1.6  Equations  to  relate  Pixel  Position  to  Angle: 

The  following  equations  may  be  used  to  convert  between  image  coordinates  and  spherical 
coordinates.  The  zenith  and  azimuth  are  in  degrees,  and  azimuth  is  clockwise  with 
respect  to  True  North  in  real  space  (not  in  image  space).  That  is,  the  bottom  of  the  image 
is  North,  and  the  right  side  is  East,  as  explained  in  Memo  AV08-037t. 

The  following  equations  are  identical  to  those  used  in  Section  2.7  of  Memo  AV09-023t, 
using  the  center  coordinates  given  in  Section  3.7  of  that  same  memo. 

<j)  =  -0.8159+  p  +  0. 003908/?  CosC^o  /  90)  -  0.0006394 p  Sin(;r^0  /  90)  (1) 

9  =  0.3357  p-  0.00002765  p 2  +  5.861  x  10  '7  p 3  +  0.0009443  p  Cos(  n  </>  /  90)  + 
0.003616  /?  Sin(  ;r  ^  /  90)  (2) 


where  (f)(]  =  arctan 


{x~xcenler) 

(k  y  center  / 


and  p  =  *J(. 


x  —  x„ 


■f  +(y~y  centerf 


(3) 


For  Site  2  data  collected  on  and  after  Feb  12  2009,  xcenter  =  255.5  and  ycenter  =  256.0. 

To  derive  the  x,y  pixel  position  corresponding  to  a  given  angular  position  9,  (f>,  in 
degrees,  the  equations  are: 


A  =  X c  +  Px  Sin( 71(f),,  / 1 80) 
Y  =  YC  +  Px  Cos{n(j)p  / 1 80) 

where 


(4) 

(5) 


P  =  3.004  0  -  0.0007256  G2  -  0.00002487  0 3  -  0.006796  0  Cos(^/90)  - 
0.02725  0  Sin(;r  0/ 90) 

<f>p  =  (j)  -  (-0.8158  +  0.003908  P  Cos(;r^/90)  -  0.0006394  P  Sin(;r^/90) 


(6) 

(V) 


A3.2.  Site  2  Night  Real  Time  Cloud  Algorithm 

This  section  documents  the  Site  2  Night  real  time  cloud  algorithm  status. 
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A3.2.1  Overview  of  Earlier  Deliveries:  The  first  version  of  the  night  algorithm  set  up 
with  the  inputs  extracted  for  Site  2  was  installed  on  March  5  2007.  The  night  algorithm 
extraction  is  documented  in  Memo  AV06-030t.  This  first  version  did  not  handle 
moonlight  well,  and  only  the  starlight  data  were  intended  for  use.  More  recently,  the 
night  algorithm  was  upgraded  to  include  moonlight,  and  the  inputs  have  been  updated. 
The  rest  of  this  section  documents  this  version  of  the  code. 

A3.2.2  Date  Installed:  The  real  time  algorithm  with  the  moonlight  algorithm  was  sent 
to  SOR  on  Feb  12  2009.  They  will  install  it  on  the  system  as  soon  as  practical.  This  has 
been  delayed  because  there  is  a  problem  with  the  link  from  SOR  to  the  WSI  processing 
computer.  Data  prior  to  this  date  will  be  processed  and  delivered  as  archival  data.  Data 
starting  on  this  date  may  be  considered  valid,  until  further  updated. 

A3.2.3  Solar  Zenith  Angles  Processed:  104  and  greater 

A3.2.4  Moon  Zenith  Angles  Processed:  0  and  greater. 

A3.2.5  Algorithm  Used:  This  algorithm  is  ProcWSID  Version  3.5.  The  updates  will  be 
documented  in  a  future  memo. 

A3.2.6  General  Comments: 

These  algorithm  inputs  are  extracted  using  data  from  15  April  -  31  July  2008.  The  data 
from  1  Jan  -  27  Jan,  2009,  have  been  tested  to  verify  that  the  algorithm  inputs  apply  well 
to  the  current  data. 

A3.2.7  Equations  to  relate  Pixel  Position  to  Angle: 

The  same  equations  used  for  the  day  algorithm,  and  provided  in  Section  1 .6,  may  be  used 
for  the  night  algorithm. 

A3.3.  Site  3  Day  Real  Time  Cloud  Algorithm 

This  section  documents  the  Site  3  Day  real  time  cloud  algorithm  status. 

A3.3.1  Overview  of  Earlier  Deliveries:  There  were  no  earlier  deliveries  of  the  day 
algorithm  for  this  site. 

A3.3.2  Date  Installed:  The  real  time  algorithm  with  the  new  adaptive  algorithm  was 
installed  on  14  October  2008.  Data  prior  to  this  date  will  be  processed  and  delivered  as 
archival  data.  Data  starting  on  this  date  may  be  considered  valid,  until  further  updated. 

A3.3.3  Solar  Zenith  Angles  Processed:  0-85 
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A3.3.4  Algorithm  Used:  This  algorithm  is  ProcWSID  Version  3.2,  documented  in 
Memo  AV08-051t. 

A3.3.5  General  Comments: 

These  algorithm  inputs  are  extracted  using  the  data  from  April  9  -  August  4  2008 
documented  in  Memo  AV09-008t.  We  were  able  to  use  the  same  inputs  for  the  real  time 
algorithm,  so  these  inputs  are  not  documented  separately. 

A3.3.6  Equations  to  relate  Pixel  Position  to  Angle: 

The  following  equations  may  be  used  to  convert  between  image  coordinates  and  spherical 
coordinates.  The  zenith  and  azimuth  are  in  degrees,  and  azimuth  is  clockwise  with 
respect  to  True  North  in  real  space  (not  in  image  space).  That  is,  the  bottom  of  the  image 
is  North,  and  the  right  side  is  East,  as  explained  in  Memo  AV08-037t.  The  equations  are 
the  same  as  those  presented  in  Section  6.8  of  Memo  AV09-023t,  and  are  repeated  below. 

The  equations  for  azimuth,  (j),  and  zenith,  0  in  degrees,  given  pixel  location  x,y  are: 

(/)=  0.6730 +  ^0  -0.0005058  /?Cos(/r^0  / 90)  +  0.007731  p  Sin(/r^0  /90)  (8) 

0=  0.3366/7  +  1.179x10- 5 p1  +  4.242xl0'7  pi  -  0.006667/? Cos(^/ 90) - 

0.0005 109/?  Sin(;r  0/90)  (9) 


where  0O  =  arctan 


(X~X  center) 
If  y center  / . 


and  p  = 


x-x„ 


■f  +(y~y  centerf 


(10) 


For  these  equations,  xcenter=  249.9  and y center  =  250.5. 

To  derive  the  x,y  pixel  position  corresponding  to  a  given  angular  position  0,  0,  in 


degrees,  the  equations  are: 

X  =  Xc  +  Px  Sin{n(j)p  / 1 80)  (11) 

Y  =  Yc  +Px  Cos{7T(j)p  / 180)  (12) 

where  P  and  (j)p  are  defined  by 

P  =  2.9800- 0.0009243  91  -0.00002075  0*  +0.051 19 £Cos(^/90)+  (13) 

0.003791 6*  Sin(;r0/9O) 

< zip  =  0  -  (0.6730  -  0.0005058  P  Cos(^^/90)  +  0.007731  P  Sin(;r0/9O))  (14) 


A3.4.  Site  3  Night  Real  Time  Cloud  Algorithm 
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This  section  documents  the  Site  3  Night  real  time  cloud  algorithm  status. 

A3.4.1  Overview  of  Earlier  Deliveries:  There  were  no  earlier  deliveries  of  the  night 
algorithm  for  this  site. 

A3.4.2  Date  Installed:  The  night  algorithm  that  handles  both  starlight  and  moonlight 
was  installed  on  October  23  2008,  and  further  updated  on  December  15  2008.  Data  prior 
to  this  date  will  be  processed  and  delivered  as  archival  data.  Data  starting  on  this  date 
may  be  considered  valid,  until  further  updated. 

A3.4.3  Solar  Zenith  Angles  Processed:  104  and  greater 

A3.4.4  Moon  Zenith  Angles  Processed:  0  and  greater. 

A3.4.5  Algorithm  Used:  This  algorithm  is  ProcWSID  Version  3.4  documented  in 
AV08-051 

A3.4.6  General  Comments: 

These  algorithm  inputs  are  extracted  using  Data  from  the  period  April  9  -  September  30 
2008. 

A3.4.7  Equations  to  relate  Pixel  Position  to  Angle: 

The  same  equations  used  for  the  day  algorithm,  and  provided  in  Section  3.6,  may  be  used 
for  the  night  algorithm. 

A3.5.  Site  5  Day  Real  Time  Cloud  Algorithm 

This  section  documents  the  Site  5  Day  real  time  cloud  algorithm  status. 

A3.5.1  Overview  of  Earlier  Deliveries:  There  were  no  earlier  deliveries  of  the  day 
algorithm  for  this  site. 

A3.5.2  Date  Installed:  The  real  time  algorithm  with  the  new  adaptive  algorithm  was 
installed  on  August  26  2008,  and  then  updated  October  14  2008.  Data  prior  to  October 
14  will  be  processed  and  delivered  as  archival  data.  Data  starting  on  this  date  may  be 
considered  valid,  until  further  updated.  The  rest  of  this  section  documents  the  October  14 
2008  delivery. 

A3.5.3  Solar  Zenith  Angles  Processed:  0-85 

A3.5.4  Algorithm  Used:  This  algorithm  is  ProcWSID  Version  3.2,  documented  in 
AV08-051 
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A3.5.5  General  Comments: 


These  algorithm  inputs  are  extracted  using  the  data  from  Jan  -  May  2008,  updated 
slightly  using  data  from  July  2008.  The  October  14  update  also  was  based  on  July  2008 
data,  but  used  a  slightly  upgraded  version  of  the  adaptive  algorithm. 

A3.5.6  Equations  to  relate  Pixel  Position  to  Angle: 

The  following  equations  may  be  used  to  convert  between  image  coordinates  and  spherical 
coordinates.  The  zenith  and  azimuth  are  in  degrees,  and  azimuth  is  clockwise  with 
respect  to  True  North  in  real  space  (not  in  image  space).  That  is,  the  bottom  of  the  image 
is  North,  and  the  right  side  is  East,  as  explained  in  Memo  AV08-037t.  The  equations  are 
the  same  as  those  presented  in  Section  9.7  of  Memo  AV09-023t,  and  are  repeated  below. 

The  equations  for  azimuth,  </>,  and  zenith,  9  in  degrees,  given  pixel  location  x,y  are: 

(j)  =  0.05787  +  fa  -  0.004426  pCo^Tifa  / 90)  +  0.0002416  p  Sin(;r^0  / 90)  (15) 


0=  0.3436/7 -0.00004412 p1  +7.145xl0'7  /?3  +0.0006305 /?Cos(^/ 90)- 
0.004436/?  Sin(^/ 90) 


where  (f)(]  =  arctan 


(X~X  center) 

(v-v  ) 

\y  y  center  / . 


and  p  =  center)2  +(y~y center)2  • 


(16) 


(17) 


For  these  equations,  xcenter  =  243.8  and  ycente=  240.5. 

To  derive  the  x,y  pixel  position  corresponding  to  a  given  angular  position  9,  (f),  in 
degrees,  the  equations  are: 


X  =  Xc+PxSin(x<f>p/ 180) 

Y  =  YC  +  Px  Cos{n(j)P  / 1 80) 

where  P  and  fa  are  defined  by 

P  =  2.932  6  -  0.0002749  02  -  0.00003042  -  0.004269  6  Cos(^/90)  + 

0.03 1 7 1  ^  Sin(;r  ^  /  90) 


(18) 

(19) 

(20) 


(j)=(j)-  (0.05787  -  0.004426  P  Cos(;r^/90)  +  0.0002416  P  Sin(^^/90)) 


(21) 


A3.6.  Site  5  Night  Real  Time  Cloud  Algorithm 
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This  section  documents  the  Site  5  Night  real  time  cloud  algorithm  status. 

A3.6.1  Overview  of  Earlier  Deliveries:  There  were  no  earlier  deliveries  of  the  night 
algorithm  for  this  site. 

A3.6.2  Date  Installed:  The  night  algorithm  that  handles  both  starlight  and  moonlight 
was  installed  on  August  26  2008,  and  further  updated  on  September  23  2008.  Data  prior 
to  September  23  2008  will  be  processed  and  delivered  as  archival  data.  Data  starting  on 
this  date  may  be  considered  valid,  until  further  updated. 

A3.6.3  Solar  Zenith  Angles  Processed:  104  and  greater 

A3.6.4  Moon  Zenith  Angles  Processed:  0  and  greater. 

A3.6.5  Algorithm  Used:  This  algorithm  is  ProcWSID  Version  3.2,  documented  in 
AV08-051 

A3.6.6  General  Comments: 

These  algorithm  inputs  are  extracted  using  the  data  from  the  period  February  1,  2007  - 
August  4,  2008.  The  data  from  July  1  -  August  4,  2008,  have  been  tested  to  verify  that 
the  algorithm  inputs  apply  well  to  the  current  data. 

A3.6.7  Equations  to  relate  Pixel  Position  to  Angle: 

The  same  equations  used  for  the  day  algorithm,  and  provided  in  Section  5.6,  may  be  used 
for  the  night  algorithm. 

A3.7.  Site  7  Day  Real  Time  Cloud  Algorithm 

This  section  documents  the  Site  7  Day  real  time  cloud  algorithm  status. 

A3.7.1  Overview  of  Earlier  Deliveries:  The  day  algorithm  was  updated  in  2005,  and 
has  not  been  updated  since  this  time.  I  believe  the  earlier  deliveries  are  not  of  interest  at 
this  time,  however  in  the  sections  below  I  will  document  the  2005  delivery. 

A3.7.2  Date  Installed:  The  real  time  algorithm  was  installed  on  10  November  2005. 
We  plan  to  extract  the  inputs  for  the  updated  algorithm  in  the  near  future,  and  will 
reprocess  all  of  the  data  up  to  that  data  and  deliver  it  as  archival  data.  Data  starting  on  10 
November  was  initially  valid,  but  we  don’t  know  at  what  rate  the  required  inputs  may 
have  changed. 

A3.7.3  Solar  Zenith  Angles  Processed:  0-79 

A3.7.4  Algorithm  Used:  This  algorithm  is  ProcWSID  Version  1.0 
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A3.7.5  General  Comments: 


These  algorithm  inputs  are  extracted  using  the  data  from  July  and  August  2005,  and  are 
documented  in  Memo  AV05-031t,  AV05-032t,  and  AV05-038t.  The  algorithm  worked 
reasonably  well  at  that  time,  but  if  the  filters  decayed,  it  would  be  less  good  in  later 
periods.  This  version  of  the  algorithm  also  did  not  include  the  adaptive  algorithm. 

A3.7.6  Equations  to  relate  Pixel  Position  to  Angle: 

The  following  equations  may  be  used  to  convert  between  image  coordinates  and  spherical 
coordinates.  The  zenith  and  azimuth  are  in  degrees,  and  azimuth  is  clockwise  with 
respect  to  True  North  in  real  space  (not  in  image  space).  That  is,  the  bottom  of  the  image 
is  North,  and  the  right  side  is  East,  as  explained  in  Memo  AV08-037t. 

The  equations  for  azimuth,  (j>,  and  zenith,  9  in  degrees,  given  pixel  location  x,y  are: 

(f)  =  -0.31 16  +  p- 0.002360/? Cos(^ ^  / 90)  +  0.00263 5 p Sin(^ ^  / 90)  (22) 

6  =  0.3400  p  —  1.3661  x  10  ^  p2  +  5.366  xlO'7  p 3  -0.003086  p  Cos(;r^/90)  - 
0.002681  p  Sin(^-^/90)  (23) 

where  (f)(]  =  arctan  and  p  =  ^{x  -  xcenterf  +  ( y  -  ycenter  )2  . 

_  y center  )  _ 

For  these  equations,  xcenter  =  247.8  and  ycenter  =  249. 1 . 

To  derive  the  x,y  pixel  position  corresponding  to  a  given  angular  position  0,  (j),  in 
degrees,  the  equations  are: 

X  =  Xc+Px  Sin{7i<t>P  / 1 80)  (24) 

Y  =  Yc+PxCos(x<f>P/ 180)  (25) 

where  P  and  (j)p  are  defined  by 

P  =  2.9610-  0.00080  d2  -  2.222  xlO'5  +  0.02259  0  Cos(;r^/90)  + 

0.01941  #  Sin(;r0/9O)  U<>> 

<f,p  =  ^  -(-0.31 16  -  0.002360  P  Cos(;r  0/90)  +  0.002635  P  Sin(^^/90))  (27) 

A3.8.  Site  7  Night  Real  Time  Cloud  Algorithm 
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This  section  documents  the  Site  7  Night  real  time  cloud  algorithm  status. 

A3.8.1  Overview  of  Earlier  Deliveries:  The  night  algorithm  currently  deployed  at  Site 
7  is  an  older,  contrast-based  version  of  the  software  that  produces  medium  resolution 
cloud  decision  imagery.  Memos  AV05-031  and  AV05-037  document  the  delivery  of 
night  algorithm  results  for  the  period  July-August,  2005. 

A3.8.2  Date  Installed:  The  real  time  algorithm  was  installed  prior  to  2005,  with 
additional  minor  upgrades  installed  on  10  November  2005.  We  plan  to  extract  the  inputs 
for  the  updated  high-resolution  algorithm  in  the  near  future,  and  will  reprocess  all  of  the 
data  up  to  the  present  time  and  deliver  it  as  archival  data. 

A3.8.3  Solar  Zenith  Angles  Processed:  104  and  greater. 

A3.8.4  Moon  Zenith  Angles  Processed:  0  and  greater. 

A3.8.5  Algorithm  Used:  This  algorithm  is  ProcWSID  Version  1.0 

A3.8.6  General  Comments:  The  contrast-based  night  algorithm  works  reasonably  well 
for  imagery  that  does  not  show  the  moon  in  the  field  of  view.  Imagery  with  the  moon  at 
medium  to  high  brightness  are  not  typically  handled  well  in  this  version  of  the  software. 
The  transmittance-based  software  will  eliminate  many  of  the  issues  found  in  the  earlier 
version  of  the  night  algorithm.  Thus  these  data  should  only  be  used  for  no  moon.  Also, 
the  geometric  calibration  has  probably  long  since  changed,  so  data  starting  around  2007 
should  not  be  used  without  checking  to  see  if  it  makes  sense  or  not. 

A3.8.7  Equations  to  relate  Pixel  Position  to  Angle: 

The  same  equations  used  for  the  day  algorithm,  and  provided  in  Section  7.6,  may  be  used 
for  the  night  algorithm. 
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